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Systems (NCHEMS). Issues involved in the development of an effective 
institutional student-outcomes database are examined, and types of 
student data, typic.ally kept in student master files and enrollment 
" files, are' identified. In addition, features of six currently 
available student-outcomes assessment instrum^ents/services are 
specified, including the NCHEMS/College Board Student-Outcomes 
Information Service, the American College Testing Program Evaluation 
Survey Service, and the TEX-:iS Follow-Up System. Attention is also 
directed to the identif icfation of users and uses of outcomes 
inf orma^tion , and the production of management reports from 
student-outcomes .databases. The application of outcomes information 
^to decision-making is also discussed, with attention to preparing 
accreditation self-studies, program review, institutional planning, 
budget review; and student recruitment and retention. Finally, some 
themes concerning information utilization are considered. (SW) 
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.0 

Until recently, the higher-education community saw littl^ 
point but no small threat in the ej^plicit assessment of student 
outcomes. The positive impact of college upon tbe.stu^cnt remained 

iiaalmQstrightccuslyunexamin.e.dp^misc— the. 

o^higher cducation.That such an attitude should persist for so long 
prnong academics and administrators is not surprising; they are, 
=^ titer all, rcarecj^in a'tradition of the value of scholarshifffor its own 
sake. Moreover, a highly supportive public ccM^tinuously reinforced 
thi^ attitu'Se d\M\ i^, ihree decades of expansion, fri turn, the public 
expected higher education to deliver on n ^lual promise— increased 
individual mobiiii/ an\! ncreased collective productivity. 

Now, of cour-^-*, things are' different. As institutional resources 
tighten, college and university administrators at all leyels are'|row 
ing moro concern'jd about identifying and improving the impact of 
their p^7o;:^ram< ipon students. One result is an escalating (though 
often leiutivcii; unexamined) interest in information on what is . 
terme^l student o\4/:comes. Furthermore, the same set of forces has 
produced a demand for greater accountability on the part of those 
controlling tl^ use of resources in higher education. More and 



more, ii^stitutions iu*c hoiilji; nskcci explicitly— and soincwiuu skcp- 
tically—to show that they arc making a cliffcrciuic. 

Despite these jt»ro\vinj» pressures, few institutions have moved 
beyond the talkii^j^ staj^e in 'the explicit assi?ssment of student out- 
ccMucs. This is all the more surprisin}^ because [is Bowen (19?7) point^f 
out, the results of such. efforts have beer generally heartening for 
those who have'tried. The main reasons tor this relucinnce are not 
hard to identify and deserve closer. examination. ^ 

^ n 

• There is a fear t\\i\t outcomes information, if collected and 

widely disseminated, will reflect badly on those collecting it. 
Partly this is due to simple lack of confidence that a positive 
5' impact will in fact be demonstrated. In greater measure, it is 
due to the f^^ir that eomplex information will be misinter- 
preted by those outside of and u^amiliar with the academic 
enterprise. 

There is a conviction that many if not most of the important 
outcomes of higher education are qualitative' and cannot 
therefore be objectively measured. Even the most reliable 
quantitative measurementof outcomes is at best only an indi- 
^ cator of the underlying attribute to be assessed. Given the 
" '^liklMilYOod of misinterpretation^^ 

see such indicators less as informational assets than as political, 
liabilities. ^ . 

• There is considerable apprehension about the "false precision" 
inherenf in quantified outcomes criteria. Regardless of their 
underlying accuracy or appropriateness, quantitative out- 
comes measures look precise. Thus such indicators tend, to 
erowd out softer forms of assessment!^ particularly in discus- - 
sions about conflicting claims on available resources. This 
tendency causes efficiency criteria to be strongly favored over 
effectiveness criteria in most evaluation processes, especially -^ 
when costs— which can be counted— loom in importance. 

Each of those reasons for shying away from the assessrnent of 
student outcomes is partially valid. Atxd all reflect the current defen- 
sive posture of higher education. Nonetheless, the use of outcomes 
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inlbriniuii^h suppDrt imcriinl dccisioi^tnnkii^i^ iuul to provielc 
./institutional accountability coiuinucs to grow. 

The fact is that foi* luost iiistituticM^s, outcomes assessment is 
now ahiuist uiuwoidahle. Amt^ig public institutions, for example, 
outcomes information is frequently required as part of mandated 
procedures for prograu^ review and prograii^ approval. In at least 
one state, explicit outcomes information is used in determining how 
a part of the state\s higher-education budget is distributed among^ 
the institutions (Bogue 1982). Among private institutions, the' 
demand for accountability, may be less insistent. But the value of 
studentroutcomes information in building effective recruitnicnfrand 
retention programs In a period of intense competition for students is 
no less C(Mnpelling a motive for collections and assessments. And in 
both kinds of institutions the role of outcomes information in 
int'cn^al budgeting processes, \h program review and unit evaluation 
processes, and in day-to-day operations hits grown immensely* An 
important undertaking in this book is thus to outline and illustrate 
some of the many ways in which institutions have used information 
on student outcomes. ^ 

But what exactly is a student outcome? While fairly well cstab- 
lishi^d in the current idi^^^ the term has been 

used by quite different people for quite different purposes. Research 
scholars in the field of higher education, for example, have used 
student outcomes to describe a wide range of phenomena, from short- 
term cognitive development to long-term changes in behavior. 
College and university administrators have used the term for the 
most'part in promoting claims of individual institutional success— 
mainly with respect to retention and graduation rates, placennent in 
advanced-degree programs, and placement in favorable employ- 
* ment situations. Student-personnel professionals have used the 
tcr^ diagnostically and descriptively to refer to the successes and 
failures of individual students, and to refer to the instruments 
needed to assess the problems and potentials of individual students. 
Most recendy, state governing boards and legislatures haye shown 
an increasing interest in outcomes as a judgmental concept— as an 
element in allocating scarce resources according to demonstrated 
institutional effectiveness. 



At IcHSi three cliflVM'ciit iipprtMK-hcs in ielLMUifyiniJi nnil inunsiiriii^ 
stiuliMit cniU'ciincs wvv present in t he htenuure, cinly one of which is 
CDinpiHible with the perspective of the ncndeiuie'ndministnitdr. * 

IVrhiips the l{ir|j[est and certainly t he oldest hody of literature on 
student outcomes treats student colle|:;e experience as the object of a 
purely (iuidcnuV /Micsti,t»(Ui()n. Numerous scholars, primarily in the 
s(H-ial sciences, have attempted to document and explain the effects 
of postsecondary education much as they would approach any other 
field of investi|i;ation/rhey look upon these effects as a body t)f data 
in terms of which to t.st and develop the theories and methods of n 
particular discipline. Two streams of develof)ment have drawn most 
attention. Psychologists have been . interested in the impact of 
college. education upon the cognitive development of individual 
students. They have concentrated on actual learninjj; gains.. as a 
result of cuurse work and also on attitudinal changes occurring as a 
result of particular sets of social experiences. Socioltjgists and 
economists have likewise been interested in investigating the impact 
of college, but primarily in terms of its contribution to social 
mobility through skills developmcnt^nnd through socialisation into 
the languages and subcultures of particular professions. 

The second major approach to student outcomes has been from 
what might be termed the studcnt'l^ersonnel per.spectii'C. Student 
outcomes are investigated as part of the process of counseling and 
advising students to select programs most suited Co their individual 
needs. Outcomes are assessed both to provide tools for classifying 
students appropriately into different kinds of treatment groups and 
to provide evidence of the degree to which particular programs are 
in fact meeting student needs. Such tools include aptitude tests for 
admissions purposes, achievcmcrtt tests for placement or certifica- 
tion of program completion, and attitudinal diagnostic tests for 
individual program planning and counseling. . 

The third approach— the management pers/jectiVc—is also the 
newest. Developed in the sixties and seventies, it views the assess- 
ment of outcomes as part of the process of rational resource alloca- 
tion and program decisionmaking. Early work at NCHEMS on the 
conceptiTal organization of the outcomes of higher education made 



hnsic con'irihutioiis i^^ ilils npproiKh, Avhich sirs in (he tlocMmciv 
tniion of outroincs im^h'ImI iiviiliijuioii K^mponcnt ofcffoctive 
pr()j:;rnin pl;innin^ iwurinidKi'tini:!. ' / 

Despite ihc fad thiu^rilUhav approaches deal with the samy. 
phemMUcnon-the iiupiUr^)f collude upon stutlcius— diffcrciKcs 
anions them enn beVonsidcrablc, To hcyin with, they have ciuite 
differeiu i;oals. Consec|iiently, they also have w|uite different dam 
rei|Uireme!Us. The primary i;oal of fhc atfjuleinic tradition is pure 
explanaiion. The object of inc|uiry is to successfully account for a 
given outcome, whetiher or not the faetoVs deemed respdnsible for 
production' of'the 'outcomes are under the control of institutional 
decisionn^akers. The major properties sought in data ftVc •inpincal 
t|uality and vajue neutrality, A^given research finding is deemed 
acceptable only if it survives conventional standardised tests of 
validity and significrtnee within a j^articular academic discipline, 
Furthermore, the finding sliould involve no preconceived nations 
abcuit the relative value of various outcomes. While the results of 
such invesftgations have been used by institutional policymakers, it ^ 
must be emphasized that cfecisignal utflity was not tWe object of the 
enterprise, 

Iw both the student-personnel and manage^^^ 
assessment7on the other hand, decisional' utility is a paramount 
consideration. The stuclent-personnel professional is not par- 
ticularly interested in the ult'mate explanation of the causes of an 
individual student problem. If a particular assessment instrument or 
technique can help effectively place a student or diagnose a problem, 
li^ttlc importance is given to whether it ultimately provides a valid 
measure of the theoretical* concept it purports to indicate. In thisr 
regard, the student-personnel prbfessionars data requirements are 
x^uite similar to those of the administrator. Nonetheless, the student- 
personnel professionals assessment usually focuses upon the indivi- 
dual student. y 

In contrast, the focus of assessment for the management operation 
is at the institutional or the program level. Because the main objec- 
tive of the management approach is improvement of the process for 
making Vesource-allocation decisions among alternative programs. 
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nppropriato dm nuist have a iuhuIht of pro|KMHii\s in addition to 
vn)|iirit*Ml viilidiiy and reliability. 

To Iv I'flivtivvly uiili^al in a nuinautMucnt setting, of anirncs 
outcomes datit, like any data, nuist be relevant, directly applicable . 
to the current inanagenieiu problem, and available when needed 
(Jnnes b\S2). The data also must have lace yalidity-tbat is, they 
must be pcrccii'a/ by decisionmakers to be (^f value in solving the 
problem.^ G(Misec|uently, the complex statistical assessment and 
analytical procedures'commiMily used by academic researchers and 
by student -personnel professi(Mials often have little impact in a 
manai^ement setting, if only because the result s of a yiven investiga- 
tion, although valid, simply cannot be. communicated effectively 
and persuasively to lay decisionmakers. 

Most available data about the impact of college upon students 
continues to be ci^rilected by means developed in the academic or 
student'|KM*sonnel ap|)roaches. Because of this, any attempt to 
measure and identify student, outcomes at tlie institutional level 
initially faces a communication task. Part of the challenge is to 
effectively demystify'the notion of an outcome, and to present it 
.clearfy from the perspective of institutional management, Once the 
concept has been properly delineated, most administrators will 
discover, as MoliereV M Jourdain discovered about prose, that they 
have beqn^'speaking it all along." Another requireirtcnt is* to effec- 
tively rc/iUc ciifferent kinds oj" outcomes and to locate them clearly 
within a specific decisionmaking context. To be effectively utilized, 
student-outcomes information must be visibly placed alongside 
such diverse elements of management information as available- 
resource indicators, activityilcvel at\d productivity indicators, and 
external policy and program constraints. The resulting array will 
provide a comprehensive picture of programmatic or institutional 
performance. Chapter 2 describes various dimensions of the 
outcomes* Concept, how it has been used, by whom, and for what 
purposes. 

Approaching the assessment of outcomes from a management 
perspective demands that the administrator carefully distinguish 
program efficiency from program effectiveness. Efficiency refers to a 
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comparison of resources expended to output produced; it is an 
assessment of thrift, waste, and prudence in'the actual production of a 
good or service. Cost per student credit hour produced is thus a 
common measure of academic efficiency. Effectiveness, on the other 
hand, involves comparison of results achieved to goals intended; it is 
an assessment of, the degree to which— regardless of cost—the 
outcomes of the production process measured up to the original 
intention. Both kinds qf assessment are of value to the adminis- 
trator, but it is important not to confuse the two. It is particularly 
important not to allow efficiency criteria to be used in judgments of 
institutional or program effectiveness. 

Other conceptual issues arise from the fact that a management 
perspective requires that the academic enterprise be expressly 
conceived of as a production process that results jn certain kinds of 
outcomes. This process operates on the raw rnaterial of its object- 
in the case of student outcomes, of course, students. They enter the 
process in a given initia] state and presumably are alteted. The 
notion o( value addec! derived from this model js much discussed and 
debated in current research about student outcomes. Essentially, it 
involves acceptance of the production-process analogy (Astin 1977). 
Student abilities, attitudes, and skills on graduation are products of 
•particular aspects of the college or university experience. In addi- 
tion, they are products of differences in the entering characteristics 
of students (differing input conditions) and simultaneous "growth/ 
maturation" experiences that students may undergo while enrolled 
but that are unrelated to the. educational experience itself. 

Two basic implications for measurement follow from this. First, 
the fact that an outcome is q change concept means that twp or more 
measurements, made with the same or similar instruments and 
techniques, are generally required. This, makes outcomes infor- 
mation mtririsically different from other kinds of, management 
inforrnation, and often, makes it somewhat more difficult to collect. 
Second, a measurement of simple change will not do. Some attempt 
must be made analytically to control for differing input conditions 
among students and fdr the different kinds of experiences they may 
undergo while enrolled. . * 
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■ The administrator's conceptual view of outcomes summarized 
abo\'e has important practical implications. First among them is the 
fact that at a given institution, student-outcomes assessment often 
involves assembling and integrating eAi5fing information more than 
collecting new information. Extensive institutional experience has 
shown that routinely collected operational data, for example, can be 
surprisingly useful for outcomes assessment. A second, reUted impli- 
cation is the importance of integrating data drawn from a variety of 
sources as fully as possible into a single student-outcomes data base. 
Such a data base, centrally located and documented, can be used 
flexibly by administrators to address a wide range of specific 
evaluative questions— from overall mission review to the assessment 
of individual instructor effectiveness. Chapter 3 describes some of 
the common types and sources of student-outcomes data available 
in most higher-education institutions. It also provides some 
guidelines on how these data can be usefully assembled in a data 
base to address specific institutional problems and processes. 

A. key purpose of such a' data base is to allow yiformation on 
student outcomes to be effectively deployed with respect to familiar 
adrhinistrative activities as well as compelling institutional issues. 
Nothing is mor.r frustrating, nor at the same time more, familiar, 
than a student-assessment or institutional-research office filled with 
neatly executed reports on student behavior— all presented 
way that the results are only comprehensible to othef 
assessment or institutional-research professionals. The trie , 
tive assessment of student outcomes, as institutions that have done 
so will attest, often lies as much in effectively communicating results 
as it does in designing and executing studies. Chapter 4 presents a 
number of utilization themes typical of different kinds of student- 
outcornes assessment efforts. In addition, chapter 4 shows that in 
order to be well used, different kinds of outcomes data must be 
clearly related to definite and, above all, accustomed administrative 
concerns. 

- As in any student-assessment effort, achieving higher levels of 
student success in and beyond academic programs is the impelling 
^ priority. In the mid-seventies, NCHEMS began its efforts to define 
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• and measure the outcomes of higher education, convinced that it- 
was impossible to improve the management of colleges and univer- 
sities in the absence of a clear definition and assessment of their 
products. This effort was grounded in a fundamental belief that 
positive impact— despite formidable measurement difficulty— would 
in fact be demt)ni,trated, and that such demonstration would benefit 

-higher -education as a - whole. As--Ben-.Lawrence, Jresident-ot:: 
NCHEMS, put it in 1977; 

Certainly it is necessary to measure impacts of higher educa- 
tion even if some embarrassments result. If higher education is 
in fact doing less than an adequate job with the resources it is 
granted, the first step toward impelling and motivating 
forward movement is to docungent the shortcomings. And 
, there will be shortcomings. But if the ability is acquired to " 
measure comprehensively the wide range of the outcomes of 
higher .education, the shortcomings will be overwhelmed by 
the documentation of an array of benefits of every sort which ^ 
now, in varying degrees, go unacknowledged. [P. 1] 

Since 1977, substantial progress has been made in our ability to 
define and assess these benefits. But far more important, substantial 
consensus has also been achieved on the imperatives involved in 
doing so and the costs involved in not doing so. As Adamany (1979) 
points out, assertions of the "unmeasurability" of academic 
programs tpecome increasingly les? credible when they come from 
institutions that cast themselves as 'social critics or whose faculty are 
themselves heavily involved in dey^eloping evaluation methods for 
other kinds of organizations. The question is clearly no longer one of 
whether, but how. "'^'J ' 



CHAPTER 2 



Conceptual 
Dimensions of 
Student Outcomes 



There are many conceptions and definitions of the term student 
outcomes as it relates to higher education. We have nq^ed 
that "a student outcome is most often defined as any change or 
consequence occurring as a;, result of enrollment in a particular 
educational institution and involvement in its programs. Within 
this broad defin-J- a number of distinctions among different 
types of outcome usefully be made. Many of these distinctions 
originated in academic research on student behavior, but some have 
their origins in the literature on effective institutional management. 

Regardless of their origins, schemes for classifying student 
outcomes generally are of two kinds. The most common approach is 
to develop a typology of mutually exclusive categories of outcomes 
that can be used as a basis for identification and measurement of 
specific outcomes. This approach has the advantage of conceptual 
clarity, but two disadvantages. It runs the risk of artificially dis- 
tinguishing things that perhaps ought to go together, and it tends to 
obscure the process elements of the educational experience. These 
objections are answered by the second approach, which is to model 
the interaction between the student and the institution as an 
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exchange process. Tlie'interaction model affords .1 dynamic view of 
what is, after all, an interchange between the student and the' 
instructor or institution. As a heuristic device, therefore, it is 
admirable. But as a mechanism for identifying and measuring par- 
ticular outcomcc, it is somewhat less useful. Whichever way the 
classificarion question is approached, student outcomes have a 
number - - secondary properties-an important one being duration. 
These secondary aspects of student outcomes must also be taken 
into account in the measurement process. In the following sections, 
the two general types of classification and the matter of secondary 
properties are considered in moderate detail. 



Some Typologies of Student Outcomes 



Perhaps the most widely recognized taxonomy of student out- 
comes is the fourfold conceptual scheme developed by Alexander 
Astin and his associates (Astin, Panos, and Creager 1967>. This'tax- 
onomy, shown in table 1, classifies outcomes along two distinct 
dimensions. The first, type of outcome, distinguishes cognitive from 
noncognitive outcomes— that is, changes in actual knowledge or 
learning on the one hand an9 changes in student attitudes or values 
on the other. The second dimension, type of data, distinguishes 
outcomes that are observable in overt student behaviors from those 
that must be identified and measured by psychometric and allied 
techniques. The differentiation undertakes neither to deny nor 
ignore the obvious^ factjhat_a%jwe3nd_cognitj^ 
complexly in^rdependent and that t^e outcomes described by 
psychological data are related in equally complicated ways to the 
outcomes represented by behavioral data. 

• Educational assessment has traditionally concentrated on the 
outcomes listed in the Psychological /Cognitive cell of the classifica- 
tion-knowledge, critical-thinking ability, basic skills, special 
. aptitudes, and academic achievement. Indeed, the measurement of 
academic achievement by grades, rank and class, and similar factors, 
produces the outcomes statistics best known and most pften used in 



12 




DIMENSIONS OF STUDENT OUTCOMES 



TADLE I 



A Typology* of Student Outcomes 



OUTCOME 



DATA 



AFFECTIVE 



COGNITIVE 



Psychological Self-concept 
Values 



Attitudes 
Beliefs , 

Drive„for Achievement 



Behavioral 



Satisfaction with College 

Personal Habits 
Avocations 
Mental Health 
Citizenship , ; 
Interpersonal Relations 



Knowledge 

Critical-Thinking Ability 
Basic Skills 
Special Aptitudes 
Academic Achievement - 

Career; Development 

Level of Educational Attainment 

Vocational Achievements: 

Level of Responsibility 

Income \ 

Awards or Special Recognition 



SOURCE: Alexander \V. Astin, R.J. Panos, and J. A.Creager, hlational hi orms for Entering College 
Freshmen— Fall 1966 (Washington, D.C.: Atnerican Council on Education, 1967); p. 16. 

colleges and institutions. General aptitude, aptitude and achieve-, 
ment in particular knowledge and skill areas, and basic-skills compel 
fence are usually-assessed by means of a variety of standardized tests. 
These tests are most often used to establish eritrance and placernent 
criteria and also to assess educational impact, or growth, up\>n 
completion:of .a particular program of study. Familiar examples 
include the College Board's verbal and math Scholastic Aptitude , 
Test (SAT) and the Graduate Record Examination (GRE). 

Psychological /Affective outcomes are somewhat more subde and 
. consequently often more difficult t o assess, but they are no less 
irhpdrtant to institutional administrators. Proponents of liberal 
education, for example , feel strongly that academic programs should 
be evaluated not only on the basis of the skills students possess upon 
graduation but also on the kinds of attitudes they hold. Indeed, 
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some of the most interesting academic investigations of student out- 
comes'are those that examined the long-term effects on attitudes ; 
and behaviors of liberal-arts graduates (Bowen 1977). Most of those 
studies report increases in such values as aestheticism, autonomy, 
and tolerance of cultural diversity. Of more immediate importance to. 
administrators, perhaps, are such factors as student self-perceptions 
""ori^me"diatTachievemonts and the so-called psychological lack of 
fit between the student and the institution. Both have been shown 
to be strongly correlated with student attrition (Tinto 1975; 

Terenzini 1979), ■ 

Behavioral /Cognitive outcomes are fnost often'assessed in terms ot 
cpreer development and subsequent educational attainment. 
Again the long-term evidence regarding the impact of a college 
education on occupational mobility is excellent (Pace 1979a). More 
immediately, two concerns have . considerable releyance when . ._ 
assessing an institution's existing programs, seeking their improve-- -r 
ment, or planning new programs.^They are the degree to which 
students are prepared to enter and progress in various occupations 
and the degree to which they are prepared for p.osrgraduate study; 
Furthermore, information on such outcomes is increasingly impor- 
tant to recruitment and public relations. 

Behavioral/ Affective outcomes probably are assessed least often ■ 
by institutioris-and probably the ones related least directly to .: 
institutional decisionmaking. For the most part, these are long-term 
outcomes, mainly interesting to researchers. Bowgn's (1977) com- 
. prehensive summary of those effects, for example, reviews the long- 
term impact of higher educatiori on political participation (more 
frequent), on consumer behavior (more efficient and informed), and 
on rmily life (more child centered, with less sex typing in family 
roles). For institutions aiming at the fUll development of students as . 
individuals and citizens, howe\'er, assessment of such outcomes 
, remains the ultimate test of success. '■ ^ 

Another well-known classification of outcomes, shown in table 2, 
was developed by Oscar Lenning and several associates (Lenning 
1977a; Lenning 1977b; Lenning, Lee, Micek, and Service 1977) at 
the National Center for Higher Education Management Systems 
(NCHEMS). This scheme is consistent with the management 
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TABLE? 

Categories of the NCHEMS Outcomes Structure 



CAT 

COOh'ff ENTITY BEING MAINTAINED OR CHANGED . 

1000 Economic Outcomes 

WOO Economic Access and Independence Outcomes 
1 1 10 Economic Access 

1 120 Economic Flexibility, Adaptability, and Security 
1 1 30 Income and Standard of Living 

1200 Economic Resources and Costs 

1210 Economic Costs and Ef.iciency 

1220 Economic Resources (including employees) 

1300 Economic Production . 

1310 Economic Productivity ond Produt^tion 
1320 Economic Services Provided - 

1400 Otlvjr Economic Outcomes \ . 

2000 Human Characteristics. Outcomes 

2100 Aspirations ■ ^. , 

2110 Desires. xAinis, and Goals 
2120 Dislikes, Lilcos. and Intei-ests 
2130 Motivation or Drive Level 
2140 Other Aspirational Outcomes 

^2200 Competence and Skills 
2210 Academic Skills 
• Z220 Citizenship and Family Membership Skills 
^- 2230 Creativity Skills 

^^2240 . Expression and Communication Skills -^^ 
2 2 50 Intellectual . Sidl]s^ 

2260" Interpersonal, LeaGlership,^d Organizational Skills 
2270 Occupational and Employability Skills, _ _ ' 
2280 Physical and Motor Skills ^ 
2290 Other Skill Outcomes 



NOTE: The fourth-level categories, into which any of the categories listed here can be divided, 
■ are "maintenance" (a fourth digit of "1") and "change" (a fourth digit of "2"). 

SOURCE: Oscar T. Lenning. Young S. Lee. Sidney S. Micek. and Allan L. Service, A Structure 
" /or - the Outcomes' of Postsecondary Education (Boulder. Colo.: National Center for Higher 

Education Management System^, 1977), p. 27. 
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TABLE 2 

Categories of the NCHEMS Outcomes Structure, continued 



C ODe'# ENTiTk^ BEING MAINTAINED OR CHANGED ' 

2000- Human Characteristlcs Outf^^^^ 

r^OO Morale, Satisfaction, and Affective Ghairactcristics 
2310 Att.:udes and Values 
' 2320 Beliefs, Commitments, and Philosophy of Life 
2330 Feelings and Emotions 
2.340 Mores, Customs, and Standards of Conduct 
23.50 Other Affective Outcomes 

2400 Perceptual^ Characteristics . . 

• 2410 Perceptual Awareness and Sensitivity 

2420 Perception of Self 

2430 ' Perception of Others 

2440 Perception of Things \. 

2450 Other Perceptual Outcomes 
.2500 Personality and Personal Coping Characteristics 

2510 Adventurousness^ and h^itiative 

2520 Autonomy and. Independence 

2530 Dependability and P.esppnsibility 

2540 Dogmatic/Open-Minded, Authoritarian/Democratic 

2550 Flexibility and Adaptability 
' 2560 Habits' ' ■ " ■ 

2570 Psychological Functioning 

2580 Tolerance^ and Persistence 

2590 Other Personality and Perscnal Coping Outcomes 

2600 Physical and Physiological Characteristics 
• .2610 Physical Fitness and Traits ■ 
2620 Physiological Health 
2630 Other Physical or Physiological Outcomes 

2700 Status, Recognition, and Certification 

2710 CompletPon or Achievement Award 

•'2720 Credit Recognition — '■ 

•2730 Image, Reputation, or Status 

. ,2740 Licensing arid Certification 

.. 2750 Obtaining a Job or. Admission to a Follow-up Program 

2760 Power and/or Authority 
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* TABLE 2 , ^ o 

= Categories of the NCHEMS Outcomes iStructure, continued 



COD M ENTITY BEING MAINTAINED OR (^HANGED ' C 

2000 Human Characteristics Outcojnes._c<j>ncm»e(:{ 

2770 Job, School, or Life Success 
2780 Other Status, Rccoj^nition, and Certification 
Outcomes 

2800 Social Activities and Roles 

2810 Adjustment to Retirement 
' 2820 Affiliations 
2830 Avociational and Social Activities and, Roles 
2840' Career and Vocational Activities and Roles ^ 
V. 2850 Citizenship Activities and Roles 

2860 Family . Activities and Roles 
2870 Friendships and Relationships 
2880 Other Activity and Role Outcomes 

' • 2900 Other Human'Characteristic Outcomes 

- 3000 Knowledge, Technology, and Art Form Outcomes . 

3100 General Knowledge and Understanding 

31,10 Knowledge and Understanding of General Facts an^J 
Terminology ■ 

* , . . 3120 Knowledge and Understanding of General Processes 
- . 3130 Knowledge and Understanding of General Theory 

3140 Other General Knowledge and Understanding . 

3200 Specialized Knowledge and Unc-rstanding 
ik 3210 Knowledge and Understanding, of Specialized Facts ... 

• and Terminology . 

3220 Knowledge and Understanding of Specialized Prpcesses'U 
3230* Knowledge and Understanding of Specialized Theory r 
" . 3240 Other Specialized Knowledge and Understanding/ 

' . ^ ^00 Research and Scholarship . _ / '^ 

, 3310 Research and Scholarship Knowledge and /v;-;;'" 
I;,"'.','. Understanding ,^::^^;i'k 

V' 3320, Research and Scholarship Products * 

>' . ' ' ' . . '''■"■^'.r: 

:■: i' ' : : :l_ : k — 
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TABLE 2 

Categories of the NCHEMS Outcomes Structure, continued 



COP E # ENTITY BEING MAINTAINED OR CHANGED , 

lOOO KhoiSv^leager Art Form Outcomes, 

continued 

3400 Art Forms and Works 
MIO Architecture 
3420 Dance 

3430 Debate and Oratory 
3440 Drama 

3450 Literature and Writing 
3460 Music , 

3470 Painting, Drawing, and Photography 
3480 Sculpture 
3490 Other Fine Arts 
3500 Other Knowledge, Technology, and Art Form Outcomes 

4000 Resource and Service Provision Outcomes 

4100 Provision of Facilities and Events 
4110 Provision of Facilities 
4120 Provision of Sponsorship of Events 

4200 Provision of Direct Services 

4210 Teaching , ^ 

4220 Advisory and Analytic Assistance 

4230 Treatment, Care, and Referral Services 

4240 Provision of Other Services 
4300 Other Resource and Service Provision Outcomes 

5000 Other Maintenance and Change Outcomes 

5100 Aesthetic-Cultural Activities, Traditions, and Conditions 
5200 Organizational Fprmat, Activity, and Operation 
5300 Other Maintenance and Change 



18 



DIMENSIONS. OF STUDENT OUTCOMES . 

i — L . ■ * r* 

pcrspecive espoused by NCHEMS in two important particulars. 
First tl/c classification undertakes to distinguish and to place along 
diffcroiit dimensions all of "the major' outputs and butcomes of 
colleges and universities as the first step toward assessiiient of 
managerial effectiveness. And second, student outcomes are not 
distinguished as such; they are delineated principally in a category" 
c/lled **human characteristic outcomes," but a few will be found 
under other headinj[js. 

The NCHEMS outcomes structure was intended to provide 
.higher education with a uniform classificatijon to facilitate the collec- 
tion of comparative information on institutional and programmatic 
effectiveness. 

' While few institutions have collected outcomes information so 
exhaustively^ or in' such detail, the classification scheme has con- 
siderable utility in assessing institutional outcomes-information 
needs and available information resources. 



Some Interaction Models of Student Outcomes 



Another well-established way of categorizing student outcomes is 
to view them in terms of the interactions among students, institu-' 
tions, and particular segments of society. Such an approach has the 
advantage 6f placi^.g student-outcomes information firmly within 
the context of other kinds of management information. This 
approach is well illustrated by the heuristic framework shown in 
figure 1, which Dennis Jones (1982) has proposed for use in structur- 
ing a management data base. 

Jones's conceptual scheme is mainly useful to test the complete- 
ness and suitability for different planning purposes of an 
institution's various data resources. =• ^ 

This framework accommodates outcomes data in two ways. First, 
^an outcome may consist of a change of state within one or more of the 
three basic and three enveloping entities depicted in figure 1. Learn- 
ing gains, attitude changes, and achievement growth are all examples 
of changes in state in the entity "students" that may occur as a result 
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FIGURE 1 

Planning Data System: The Central Entities 




SOURCE: Dennis P. Jones, Data and Information for Exeaitive Decisions in Higher Education 
(Boulder, Colo.: National Center for Higher Educapon Management Systems, 1982), p, 38. ■ 

of an educational program. The sec6nd form that an outcome may 
take is that of a change in relationship between two entities; If a \ 
student drops out of school because of a negative experience, a 
change in relationship takes place between the studentand the insti- ■ 
tution. If a student finds a job as* the result of completing a certifica- v ' 
tion program, a change in relationship is registered between the f 
student and a particular critical constituent— the employer ■ " 

The key advantage of Jonese s interaction model4s that it classifies ; 
^different kinds of management information in comparable forms. 
Information about students is only one portion of the inform'^ation • 
needed to inform m^anagement decisions about progi^ms— for 
instance, whether to mount a new degree program. Information also ;^ 
is needed on external support for the program, which involves the | 
relationship between^^p ins'ti^iution and employers; on competitors h 
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that offer the same or a similar program, which involves data on the 
characteristic^ of other providers; and on the current ability of the 
institution to offer the program, which requires ^ata on the 
resources of the institution. The framework reminds thelidminis- 
trator that information on student outcomes appropriately infprms 
"decisions only when it is in the company of other kinds ot intor- ~ 
mation. And the framework helps the administrator identify' the g 
various kinds "of information that are' relevant to the decision 
at hand. ^ 

Among the various categories of student-outcomes information, 
that concerning th^ 'relationship between the student and the - 
institution is probably most important. It can be modeled as an 
interchange: institutions utilize resources to deliver educational pro- 
grams targeted to various groups of students. For their part, students 
choose programs that enable them to fulfill important personal 
objectives. Jones*s interaction model highlights the fact that both 
the institution and the student are conscious, goal-oriented actors, 
each of whose behavior influences the other. We are quite used to 
modeling institutions ij\ this way; such terms feis mt55ion, program" 
rnatic goa/s, and resource constraiht are freely used in institutional 
planning and*decisionmaking. We are less accustomed to thinking 
about students in these terms. . 

Yet it is critically important to remember that students as well as 
institutions have programs, if what we mean by program is a fairly 
well defined set of goals, a set of behavioral objectives, and a'set of 
strategies to gain these objectives withitf-the limits imposed by the 
resources available to the student. If a student's program, thus 
defined, is not often as well articulated as ^hat of an institution, at 
least sometimes it is. (In fact, educators from time'to time question 
the wisdom of early career decisions, made in the high-school years, 
frequentl y by the brig htest st udents, and c o mpulsively__ pursu ed 
without a second thought right through college and graduate • 
programs.) However well students may know their own intentions, 
institutional administrators generally know only a part of the 
student's overall program — the part that directly involves the insti- 
tution. The student qualifies for, elects, and pursues a particular 
program, and the institution ihfers that program completion is" the 
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student's main goal. The concept of attrition provides an excellent 
illustration of this myopia. From the point of view of the institution, 
attrition is an unsuccessful programmatic outcome. In fact, often it 
may be ai^succe'ssful outcome from the point of view of the student: 
the relationship with the institution was terminated because the 
student's educational goals had been fulfilled, and program comple- 
tion was not one of them. 

So another positive consequence of the interaction-model, 
approach is that it keeps the administrator aware of important 
distinctions between programmatic outputs and their associated 
outcomes. The institution produces a multitude of outputs through 
its many programs and services directed to students. These outputs 
-range from individual classroom experiences to student-life experi- 
ences and informal uiteractions with faculty. The combined effect of 
the particular mix of outputs associated with a particular student 
produces a body of educational outcomes for that individual. Assess- 
ing the comprehensive cause of a particular student outcome-a 
change in enrollment status, employment status, or attitude-is 
thus more than just empirically difficult. The discrete outcomes 
" represented in data are not so separable in the student's mind. They 
are embedded in a holistic felt experience; even a §.tudent with 
marked gifts for introspective insight and causal analysis would have 
gr-at difficulty cutting through the tangled interrelations of the 
various outcomes and isolating them. Researchers are recognizing 
this increasingly,, as evidenced by a growing number of 'nvest.ga- 
tions of campus "environments." (See, for example, Moos 1979^ 
Ultimately, of course, one can only hypothesize with respect to such 
experiential phenomena. But the accepted wisdom among those 
who specialize in student-outcomes information holds that the attri- 
bution of cause can only be to the whole body of experience the 
student undergoes-the rccumulatcd outputs of many programs, 
acting in combination. 

Furthermore, the outcomes of the same combination of institu- 
tional programs may be quite different from one student to another. 
The administrator is strongly tempted to conceive of outcomes as 
changes in average student performance or attitudes. But this can be 
a highly misleading not ion. As Bowen has pointed.out, few studi es 
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of student outcomes have demonstrated any marked changes in 
average student characteristics or behaviors as a result of college 
experiences. This does not mean, however; that individuals in the 
population of students represented by the average have not under- 
gone substantial changes, both positive and hegaiive (Bowen 1978). 
This should not-deter institutions from compiling and interpreting 
statistics on average student changes. But administrators should not 
lose sight of the fact that the individual student constitutes the core 
ofjthe concept of student outcomes. Common denominators are no 
more than shadows. 



Some Additional Dimensions of Student Outcomes 



Typologies and interaction models both provide starting points 
for sorting out the many sorts of phenomena that can be grouped 
under the heading "student outcomes." But a.number of additional 
outcome dimensions cut across all such classifications. These extra 
dimensions are useful in distinguishing among particular outcomes 
of the same general type. 

• Outcorries may be either shoruerm or long'term depending on 
their persistence over time and on the length of time after 
graduation or program completion it takes for the outcome to 
appear. Long-term persistent outcomes often are claimed as a 
major benefit of ' higher education. Most administrators, 
though, are preoccupied with the assessment' of short -term 
outcomes. This focus of attention is appropriate for two 
reasons: k is difficult to. collect data on long-term outcomes, 
and they rarely can be shown to be linked back to explicit out- 
puts over which institutional decisionmakers and managers 
have some measure of control. (The links may be there, but we 
do not have the analytical means to make them indisputably 
visible.) Assessment and communication of long-term out- 
comes, however, probably has the greatest potential for 
influencing public perception of the value of higher education 
as a whole. 
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• Outcomes may be sub ject-aware or sub ject-unawarey depending 
on the degree of the student's consciousness of the outcome. 
For example, a student's own perception of growth in a par- 
ticular academic area may have little relation to actual levels of 
achievement, although these factors have been shown to be 
generally correlated (Pace 1979b). More important, students 
may be quite unaware of changes in their values; retrospec- 
tively, they find it hard to believe (or to admit) that they ever 
felt differently. Again, administrators will be primarily con- 
cerned with subject-aware outcomes, because they are rela- 
tively easy to measure and because they are most directly 
related to student choices of institution and program, level of 
effort, and persistence. 

• Outcomes may be either direct or indirect, depending on how 
closely they are causally connected to an educational program. 
A student's acquisition of new facts and skills as a result of 
attending a particular course is a direct educational outcome, 
A graduate's persistent interest in civic affairs, stimulated in' 
part by participation in student government while in college, is 
an indirect outcome. Both direct and indirect outcomes 

' should concern administrators. Because indirect outcomes 
usually are more difficult to identify and measure, adminis- 
trators should be particularly sensitive to the variety of in- 
direct influences their institutions may have upon students, 

• Particular outcomes may be cither intended or unintended con- 
sequences of various programs.' An example of an unintended 
consequence of a two-year baccalaureate transfer program at a 
community or junior college might be a successful transfer to a 
four-year institution well before the two-year program could 
be completed. While administrators properly concentrate On 
the. assessment of intended outcomes, a sensitivity to 
unintended outcomes can yield quite unexpected information. 
In the case above, for example, the institution was experienc- 
ing a negative outcome because it was, in a sense, too successful 

Many similar distinctions can be drawn among different types of 
student outcomes. Some, like the earning of a degree, are easy to 
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measure. Some, like the ability to think critically, are more difficult 
to measure. Some outcomes involve changes in knowledge, skills, 
attitudes, or activities; others preserve knowledge, skills, and so on. 
This book contains a bibliography of recent significant literature 
on the conception and assessment of student outcomes. While each 
of the approaches and classification schemes described in this 
literature has its virtues, the administrator should approach the 
delineation of outcomes flexibly and creatively. Classification 
schemes of every kind are better employed by administrators as 
guides to thinking about needed information* and" available data 
than as rigid prescriptive devices for determining a datarcoUection 
plan. 

It should be remembered, too, that the value of information about 
any^ given outcome depends on the goals and objectives of the insti- 
tution or program, whether or not it is fashionable in academe. 
Most administrators, for example, share in the wide predisposition 
in higher education -to value a traditional set of outcomes of the 
undergraduate college experience— those associated with liberal and 
• general education. Administrators of regional universities and com- 
munity colleges alike may see an equal need to stress outcomes such 
as job placement and the development of occupational skills. If 
different from the traditional academic outcomes, these more 
mundane outcomes are not thereby necessarily degrading to the 
traditional academic values espoused by colleges ar>d universities. 
The new diversity among institutions as well as among students 
need not threaten the sanctity of the traditional academic enterprise 
so long as we recognize that educational gains can legitimately 
occur— and be effectively measured— along a number of different 
dimensions. 
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Building a 
Student'Outcomes 

Data Base 



Effective use of studentoutcomes data hinges on adequate 
integration and meaningful interpretation of data drawn 
from a wide variety of sources. Furthermore, usefully informing 
administrative decisions will often require the integration of 
student-outcomes data with other kinds of management data— for 
example, productivity data and resource data. So it is crucial for 
those trying to use outcomes data (or any other kind of data, for that 
matter) to fully understand a-material distinction between data and 
information. 

Data (for example, the responses to a questionnaire or the entries 
in a course-registration file) are the direct results of observation or 
measurement. They are the raw facts from which information can 
be constructed. Information, on the other hand, consists of data that 
have, been selected, combined, and put into a form that conveys to a 
given recipient some useful knowledge uport which to base action. 
The quality of data , is determined by the validity, accuracy, and 
reliability of the measurement process used in data collection. The 
quality of information is determined as much by' its relevance, 
timeliness, arid acceptability to potential users as it is by its deriva- 
tion from fiigh-quality^data Qones 1982). ■ 
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TABLE 3 

, Student Master-File Data Elements Useful in Conjunction with Survey Data 



Academic DackKround/Aptitudc 


Personal Background 


Enrollment Data 


Academic Record 


Predicted CPA 


Sex ' 


Student Type/Scatus 


C'-editb Attempted (by term) 


High School. Attended 


Age 


Original Entry Date 


Credits Earned (by term) 


High'School Class Rank 


Raee/Ethnic Group 


Original Entry Status 


Term/Cumulative GPA 


High-School CPA 


Residence 


Day/Evening Enrollment 


CEEL Credits Granted 


High-School Graduation Year 


Citizenship 


Residence Hall/Status 


Probation /Suspension Status 


Advanced-Placement Credits 


Veteran Status 


Major/Minor 




Transfer College 


Marital Status 


Advisor ■ 




Transfer OPA 


Financial-Aid Status 






Test Results (SAT, ACT, etc.) 








Placement-Exam Results 









32 



STUDENT-OUTCOMES DATA BASE 



Because users define information in different ways, the actual uses 
.of information and consequently the forms in which it is needed are 
variable— and Irequendv the variability is not predictable. The con- 
cept of a data base seeks to ameliorate this difficulty by clearly 
distinguishing a set of ongoing data resourci^s for the many and 
varied uses to which derived information may be put. If developed 
properly, data bases can be stable over time, while the array of uses 
and users of information may change. 

A major concern in constructing a studentoutcomes data base is 
the choice of what kinds of data to include and what kinds of general 
mechanisms and formats to employ in converting data into different 
kinds of information. An initial data base may contain only a set of 
reports from different student follow-up surveys or studies, kept 
together and documented for ready reference to respond to par- 
ticular questions from administrators as they arise. Many institu- 
tions have greatly strengthened their ability to address outcomes 
issues simply by preparing an inventory of available data, together 
with materials showing how the data were collected, by whom, 
when, and for what purpose. A data base could consist of the 
responses to different studentoutcomes questionnaires, archived in 
machine-readable form. Or it might consist of questionnaire 
responses combined in a single computer file with other individual 
student data drawn from enrollment and master-file data. Table 3 
summarizes some important types of student data, typically kept in 
student master files and enrollment files, that can be used in 
conjunction with the results of student surveys. 

1. Institutional Sources of Student-Outcomes Data: 

As table 3 suggests, one of the most important steps in building an 
effective institutional studentoutcomes data base is to recognize 
what kinds of data are already available about students in different 
parts of the institution. Many administrative units and academic 
departments in colleges and universities collect data on student out- 
comes. Generally, however, these data will be scattered throughout 
the institution, having been collected by different administrative 
units at different times, by different methodologies, and for different 
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TABLE 4 

Types and Sources of Outcomes Data 



Student Characteristics 



Relationships 



FROM OPEU ATING SYSTEMS: 

Sex 
Age 
Race 

Residence 
CPA 

(Family-Income Class) 
, (Religious Preference) 
(Test Scores) 



FROM "SPECIAL SOURCES": 

Employment Status 
Marital/Family Status 

(Income) 

Educational Goals 
Educational Aptitudes 
Perceptions of Gain 



FROM OPERATING SYSTEMS: 
Level . 

Program Category/Status 
Full Time/Part Time 
Day/Evening 

Load (Sch/Contact Hours) 
GPA/Probationary Status 
Degree Status 

(Placement) 
(Alumni Activities) 
(Certification) 

FROM "SPECIAL SOURCES": 

Employer/Industry/Job 
Satisfaction with Training 

Subsequent Education 
Satisfaction with Preparation 

Perception of Services 

Certification (External) 

General Education 

Subsequent Achievement 
(Professional/Civic) 
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purposes. Table 4 lists kinds of studcnt<lata elements often collected 
by. colleges and universities. The table breaks down these data 
elements according to whether or not they are derived from regular 
institutional sources or special surveys, and according to whether 
they primarily have to do with student characteristics or with the 
relationship between a student and the institution or another 
constituent. 

Few colleges and universities will have collected consistent and 
accurate data on all of these elements. But few institutions will have 
no available information on most of them. The most complete data . 
will probably be. that whicji is regularly collected on student charac- 
teristics. The majority of these data will be found in a student-records 
file, generally maintained by the registrar. Occasionally, such data- as 

. family-income class or test scotes will be available on each student or 
on selected students but will have been collected and will reside in 
different locations, such as the financial-aid office or counseling 
center. Most of these data do not deal with outcomes directly. 
Rather, , they are useful for delineating particular categories of 
students among which outcomes comparisons may be drawn. 

Most colleges and universities will also have a good deal of 
routinely /collected data on relationships. The bulk is standard 
enrollment data. Placement, certification, and alumni data may be 
available but usually on certain students only and then often only 
through particular offices. Nevertheless, when combined with the 
student-characteristic data available through student registration 
systems, this array of data elements alone provides a powerful 

- resource. Enrollment and student<haracteristic data, for (example, 
constitute a foundacibri' for constructing a student-tracking data 
base that can generate extremely useful sets of fundamental out- 
comes studies. Attrition and retention rates, for instance, may be 
calculated and ^compared for different categories of students. 
Changes in course-taking patterns and course<ompletion rates can 
be" similarly analyzed. Such studies usually 'are. relatively un- 
sophisticated and generally are confined to assessment of a fairly 
limited set of outcomes. But often they are the most immediately 
useful student-outcomes studies an institution can engage in. Even if 



31 



. TABLE 5 

Existing Outcomes Data Commonly Collected on Campus 



Source/ LiKotion 



Academic Dcpts. 
Student-Services Offices. 

Alumni Office 
President's Office 
Placement Office 
(System Offices) 



Type* of Data 



Follow-Up Studies 

Program Accreditation Reports 

Service Evaluations 

Exit Interviews 

"HiglvRisk Student" Studies 

Alumni Activity Surveys 

(Alumni Lists of Current Addresses) 

Accreditation Self-Study Reports 
Ad Hoc Studijcs 

Placement Surveys 
Employer Surveys 

Occupational Follow-Up Studies 
Employer Surveys 
Transfer Tracking Studies 



these are the only data elements available to an institution, the infor- 
mation they yield can be impressive ^and of immediate policy valine., 
(See, for example, Ohio State University .19131). . 

Fortunately, other kinds of sporadically collected data on student 
outcomes are also commonly^' available at most institutions— 
although scattered widely throughout the campus. Table 5 lists data 
typically collected, together with the offices generally responsible for 
their collection. Such a list is a good place to start in inventorying an 
institution's current student-outcomes data resources. 

Again it should be emphasized that the data collected by these 
different efforts will vary widely in quality, will have beerv assembled 
at different times, and will cover different categories of students. If 
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collected by student survey, Avhich is the most common method, 
these kinds of data share a number of* important limitations. of 
which administrators should be aware. 

One set of limitations has to do with the problems of ^'response 
bias" associated with most special studies and surveys. Because of 
cost, mat^y surveys are administered to only a small sample of the 
population to be assessed. In itself; sampling causes few problems. 
But response rates of 100 percent are rarely attained, and there is 
seldom .any assurance that those who did not choose to respond 
are substantively the same as those who did. Alumni surveys-— 
especiially if conducted independently by alumni offices for their 
own purposesr-are particularly pronip to bias because of selective 
response. Many institutions have published reports based on 
response rates as low as 10 to 15 percent; response rates of 30 to 40 
percent are probably typical of such efforts. While bias of this kind 
may not render survey results completely unusable, it does require 
the administrator to take into account the characteristiig^ of.those 
responding before accepting their responses as typical. 

Another set of limitations has to do with the inherent instability 
of student self -report data on attitudes. Attitudinal responses may 
change markedly over the course of a,.student*s enrollment as a 
result of different experiences encountered or progress attained. A 
number of studies repeatedly 'assessing student attitudes have found, 
for example, that students increasingly value general education as 
the length of the enrollment increases (Feldman and Newcomb 
1969; Terenzini 1983). It is generally impractical for most institutions 
to survey the same students repeatedly. Some caution must be 
taken, therefore, in interpreting data on attitudes that may already 
have changed as a result of a new institutional policy or program, or 
simply because the student has progressed. 

A third set of limitations has to do with response distortion- 
deliberate or unconscious— on the part of student or alumni 
respondents. For example, jt is fairly common for a large proportion 
of students surveyed when withdrawing from college to report that 
their primary reason for doing so is financial pressure. Face-to-face 
interviews with these same students will generally reveal, however, a 
much more complex motivational picture-~one in which financial 
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pressure. plays little or no part (Lenning, Beal, nncl Snuer 1980). 
Careful questionnaire design or study^administrntion procedures 
Kan help avoid this problem. But again, it is important for admiuis' 
trators drawing implications from data qf this kind to be aware o( 
the possibilities of distortion. 

Because of these limitations, no single study or source should 
stand alone in an institutional outcomes^assessmcnt effort. 
Together, however, such a body of sources will often yield n con* 
sistept body of findings. Indeed, a major objective in building an 
outcomes data base is toprovide as many inde/JL'ndent sources of data 
as possible. An assortment of findings, if they are consistent, 
provides considerable confidence' that underlying patterns of 
outcomes.are present and that the information may safely be used as 
a basis for administrative decisions/ 

2. Identifying the Users and Uses of Outcomes Information: 

The desirability of institutionwide.involvemcnt in the design of a 
student-outcomes data base cannot be overstated. A number of 
different institutional personnel and constituencies may have an 
interest in the kinds of information contained in such data bases; 
many also have particular pieces of information to contribute or 
may control the sources of particular kinds of data. Involving these 
parties at interest in the early stages of planning the data base 
maximizes the possibility that the available data will.be effectively 
used. It also enhances the li{<e!ihood that atl offices and individuals 
will cooperate in any future data-collection effort. 

One vehicle for involving institutional staff and other interested 
groups is an advisory committee comprising students, faculty, and 
administrators. A committee of this type can make recorhmenda- 
tions concerning the administration and use of existing student' 
outcomes information, It can also suggest additional surveys that 
ought to be undertaken to enrich the data base. Representatives of 
student services, alumni development, admissions, career place* 
ment, vocational administration, data processing, and public infor- 
mation may all share th^ interest of academic administrators in the 
findings of student-outcomes surveys and in the implications of 
previously collected data. 
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The involvement of n number of In'slitutional personnel will also 
help to RUard agninst overlapping activities in different offices and 
departments. In many institutions, large ones in particular, redun- 
dant surveys are often conducted, at a waste of time and money. 
Early involvement of these offices in survey planning will not only 
help forestall such duplication in the future; it may also uncover the 
findings of past surveys of which other offices mny be unaware. ,, 
.An additional benefit of an advisory committee composed of^^ 
interested parties is that the members may begin to use in their own'>' 
units the kinds of information they obtain through committee par- 
ticipation. As they attempt to directly apply available iriformation, 
and the itisights derived from this information,,to the problems and 
tasks in their own units, committee members will be testing the 
utility of different ways of presenting and integrating dnta drawn 
from an institutional student-outcomes data base. At the same time, 
they constitute an initial body of trained data users, able to pass on 
their skills to other administrators. Indeed , a priority function of the 
committee should be to make administrators aware of the relevan?e_ 
of outcomes infoj-mation to the day>to<lay operations of the units 
and departments they manage. 

One task involved in ensuring that the mform'ation is relevant is 
to identify key users of information and the levels of. information 
used throughout t-he institution. By the time an institutional 
advisory committee is established, this task may already have been 
partly accomplished. Thekinds of questions and issues' raised by the 
advisory committee should provide an outline of critical issues that 
an institutional student-outcomes data base should address. In any 
case, it is likely to be most useful to sur^imarize these assessments in 
• terms of an explicit set of research questions for each user or use that 
the information provided I ■ the survey may be able to answer. To 
illustrate, the dean of student services might ask: "What are the 
characteristics of those who are sadsfied and dissatisfied with 
particular services?" And an academic dean might want to know: 
"What are the differences among departments regarding the 
academic and personal goals of their students?" or "Which depart- 
ments have been most effective in placing their students in jobs 
related to their training or in programs of further study?" 

^ ' 35 
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Similarly, research questions associated with an identified institu- 
tional problem, such as attrition and retentionv might include: 
"What are the characteristics of students who drop out as opposed 
to those who persist?** or "What particular negative assessments of 
the institution and its services are given by those whoJe^e?'* or 
'*'Are the academic and personal aspirations of those who leave 
different from those who Stay?" 

One useful device at this point in an outcomes-data planriirfg 
process, especially in structuring committee discussion, is to 
construct a so<alled source/use matrix of available information. 
Such a matrix is illustrated in lable 6. it provides a convenient way 
to summarize existing institutional outcomes data resources and 
to identify the relevance of this data to recognized institutional 
problem areas., A source/use matrix will also help locate gaps in 
available outcomes data. 

In completing such a matrix; and indeed in any data-integration 
activity, the role of the institutional-research or similiar office may 
be ^critical. To begin with, the ^central location of such an office 
makes it an appropriate agency for aggregating and cataloging exist- 
ing information on student outcomes. More .important, the consti- 
tuency for this office is generally the institution as a whole. This 
"makes it an appropriate place notpnly for aggregating information 
from various sources, but also for integrating such information in 
terms of common institutional issues or problems. 

Even when particular kinds and sources of outcomes data are 
known to be available, however, they are frequently not in the 
"proper format to be of immediate use to decisionmakers. The 
primary method most offices use to disseminate studentoutcomes 
data is a standard-format report on a particular data-gathering 
effort. For example, each time a particular office completes an 
alumni follow-up study or a tracking study, the results are reported 
independently. Such reports can be said to be methodology driven 
rather than problem driven. Thus one of the first tasks for an effec- 
tive utilization effort is to reorganize thc' information contained in 
several published reports so that it clearly relates to to problems 
faced by institutional decisionmakers. 
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Subtle and sensitive interpi^i^tation is sometimes required to effec- 
tively integrate fragments of information— often by themselves of 
little significance— into the single coherent picture of a phenomenon 
that an administrator needs. One such exercise, assembling the 
results of numerous attitudinal studies of currently enrolled 
students and integrating these with tracking^study results on the 
attrition patterns of different kinds of students, has been particu- 
larly effective. However, much of the process of reorganizing existing 
material merely involves simplifying data presentations to highlight 
major trends, to emphasize comparisons, and to eliminate extra- 
neous material. The continuous involvement of an institutionwide 
committee that can bring multiple perspectives to bear throughout 
this process is greatly valuable. 

As with any other* creative planning-oriented exercise, no one 
best way will be found to bring existing institutional information on 
student outcomes to bear on identified campus problems, Nevvr- 
theless, it is good to involve as many key users ancl generators of 
outcomes information on campus as possible, throughout the 
process. Beyond the identification of particular bodies of informa- 
tion relevant to common problems, such an exercise will often be of 
value in simply gathering diverse groups of administrators and 
faculty for uncommon opportunities to talk about student 
experience. 

3,!Designing Additional Efforts to Gather Outcomes Data: 

If such a device as the source-use matrix can be useful in breathing 
new life into old data, it may well be critical in designing procedures 
for gathering new institutional student-outcomes data, A primary 
difficulty with most institutionally based efforts to collect student- 
outcomes data is that they are planned and executed in an atmo- 
sphere dominated by technical and methodological considerations. 
Because such efforts generally involve cdnstruction^of a valid survey 
strategy, technical considerations are indeed imiportant. After an 
initial planning, meeting to set the goak of the survey effort, the 
technical details usually are left to technically oriented staff, such as 
institutional researchers or student-personnel professionals, to work 
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out. As a result, survey-based data<ollection efforts carried out by 
research professionals have a strong tendency toward being 
technically exemplary but operationally useless. 

The tendency of the administrator, of course, is diametrically 
opposed. Unchecked, the administrator will demand perfect instru- 
ments, which will give direct answers to immediate questions. 
Committees composed largely of faculty members without substan- 
tial survey-research experience exhibit perhaps the strongest drives 
in this direction. - \ 

Striking a .balance between technical feasibility and policy utility 
is best accomplished by establishing consistent and regular channels 
of communication. One reason why institutions should carefully 
assess their existing student-outcomes data resources, as described in 
the previous section, is' that the process promotes concrete discps- 
sidn of the technical adequacy and policy , utility of existing instru- 
ments and techniques. Because of its concreteness, such discussion 
can be infinitely more satisfying and of much greater usefulness than 
a discussion of a hypothetical future data-gathering effort. 

Indeed, one technique that has proved to be valuable in institu- 
tional settings is to ask committee members and managers what they 
expect existing data to show— and why— before they actually, see the 
results. This forces concentration on the utility of the instruments 
and techniques in use, in addition to exposing initial assumptions 
about student growth and attitudes. When the data actually are 
made available, comparing actual results with expected results can 
be beneficial and revealing. 

If additional student-outcomes data is deemed necessary, many 
institutions will decide to develop a data-gathering strategy from the 
ground up, involving construction of an institution-specific survey 
questionnaire. and a method for its administration. This is appro- 
' priate in 'many cases. Nevertheless, administrators should be aware 
of the wide and growing range of well-constructed and proven 
sy,stems and questionnaires for gathering student-outcomes data. 
These, instruments are generally far superior technically to instru- 
ments developed locally. They also will provide data compatible 
with data from other institutions i!ising the same instruments. Of 
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TABLE 7 

Some Features of Currently Available 
Student-Outcomes Assessment Instruments/Services 



The NCHEMS/CoUege Board Student-Outcomes Information 
Service (SOIS) 

—Designed to assess„student attitudes, perceptions of growth, and 
subsequent educational and employment experience at different 
points of a student's college career. Includes evaluations of specific 
college services. • ' 

—Contains six questionnaires in two-year and four-year versions: 

• External-student questionnaire 

■ • Continuing-student questionnaire 

• Former-student questionnaire 

• Program-completer questionnaire 

• Recent-alumni questionnaire ' 

•. Long-term alumni questionnaire ' • 
~A computer data-analysis service is available. 
—Comparative data from others using the service are ^available by 
type of institution. ' 

— Additional local questions may be addeduo standardized 
questionnaires. ' 

The ACT Evaluation Survey Service (ESS) 

— Designed tb assess student attitudes, perceptions of growth, and 
subsequent educational and occupational experiences at different 
points of a student's college career. 

^ —Includes three survey questionnaires: 

• The student opinion survey , 

• The withdrawal/nonreturning student survey 

• The alumni survey . ^ \^ 

— A computer data-analysis service is available. 
—Comparative data from others using the service are available by 

type of institution. \ 

— Additional local questions may be added to standardized 
questionnaires. 
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TABLE 7 (continued) 



The TEX'IS Follow^ Jp System 

—Designed explicitly Un v'. 'nmunity and two-year colleges. 
—Designed priomarily to asst. s occupational, employment, and 

continuing-education outcomes. 
—Includes seven postciv u ^zed questionnaire instruments: 

• Student's educdtlunal intent 

• Withdrawal follow-up 

• Nonreturning-student follow-up ^ 

• Graduate follow-up 

• Employer follow-up 

• Adult and continuing-^ducation follow-up 

• State follow-up reporting 

—Designed explicitly to maximize mailed survey response. 

The Cooperative Institutional Research Program (CIRP) 
Student Information Form 

—Designed for use in "value-added" research. 

—Contains a wide variety of student goals and general attitudinal 

items. ■ , ' 

—Comparative data across years is available, r 
—National norms are published by type of institution. . , 

The ACT College Outcomes Measures Project (COMP) 

. —Designed, to measure and evaluate general knowledge and skills ' 
presumed to be outcomes of undergraduate education. 
—Contains three instruments for assessing six areas of general 
knowledge: • - 

• Objective test 

• Activity inventory 

• Composite examination 

—Involves student oral and written presentations as well as 
standardized response formats*. 

The Pace College Student Experiences Questionnaire (CSEQ) 

^ —Designed to measure "quality of student effort" in college. • 
—Contains fourteen' scales/measures on the use of college facilities 

and opportunities for learning and development. , 
—Contains eight scales/measures on student assessment of the . 

college environment. ■ 
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course, such instruments may not cover particular data^elements 
deemed important by the^ institution. Most, however, include 
"an "additional questions" 'section in which institution-specific 
items rnay be locally developed and added to a standardized survey 
questionnaire. 

Several of these readily available student-outcomes assessment 
systems are briefly described in table 7. All are worth a careful look 
by administrators as well as technical personnel, if institutionwide 
outcomes-data collection is being seriously considered. (The chart in 
the appendix ties each of these instruments to particular student- 
outcomes dimensions and may be used as a preliminary guide to the 
.kinds of outcomes emphasized by each instrument.) 

The bibliography provides an.i ample selection from which. to 
choose a technical guide for assessing student outcomes or designing 
student-outcomes data-gathering efforts. The \yhole. process 
obviously cannot be discussed here in detail. The main point t6 
remember is that the administrator is responsible for. seeing that 
existing data. are appropriately inventoried and orgariized and that 
future data-gathering efforts are appropriately scaled to institutional 
requirements and directed at institutional problems. 

. 4. Generating Management Reports from Student-Outcomes^Data, 
Bases: 

Regardless of its scope or sophistication, the purpose of a student- 
outcomes data base is to serve as an ongoing, continuously updated 
information resource to answer administrative questions about 
program effectiveness as they arise. Perhaps the greatest advantage 
of an ongoing student-outcomes data base is that it readily allows 
comparisons to be drawn among a wide variety of different pro- 
grams, institutions, or student groups. Much of the artpf converting 
questionnaire data into usable institutional information lies in 
building effective comparatwe reports and ■data-presentation 
formats. Table 8, for example, succinctly summarizes, for^ broad 
university constituency, many of the main points of four successive 
administrations of a standard entering-student questionnaire. 
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Entering-Student Questionnaire Sdmmary 
1978'-1981 



% Female 
% Black* 
% Age 21 + 
% Not seeking BS 
. % Seeking business degree 
% Who say following reasons influenced enrollment decisions: 
Guidance counselor 'advice was very important 
TSU, recruitment was very important 
Relatives were very important 
Teachers*, advice was very important 
■ TSU academic reputation was not impo^tar]^^l.^:;vi(^l|^S^ 
. % TSU first choice ^ ' ^ .r^^-^^^L^v.-^^^^^^^ 

, % Who plan to work .d^'lk'^^'^P^ 
^S() Not needing'btcupational 

%-ilNjototieeH][f?f^^^ ' 
rr^^'9^'1Sareer planning finds good job fit 
% -Wanting placement office job searches . . , 

% Not using private car 
% Not from single-parent family 
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The list of items included in this example is far from definitive and, 
should be modified to suit the assessed needs and interests of admin- , 
istrators at a particular institution. If student master-file data are' 
available to supplement results derived from questionnaire surveys, 
a number of additional items can be usefiilly added. These include 
such summary characteristics of entering students as SAT/ ACT 
scores,^verage high-school CPA, and percentage receiving financial 

aid. ' - : ' 

A second useful kind of comparison is of student-outcomes survey 
res^^^^ institutions. Some widely used outcomes 

questionnaires— among them the Student Information Form of the 
Cooperative Institutional Research Project . (CIRP) and the in- 
struments of the NCHEMS/College Board Student-Outcomes 
Information Service (SOIS)— provide summary results of student, 
responses from the various types of institutions that have partici- 
pated in the survey; These kinds of comparative analyses are limited-, 
of course, by the number of institutions that have previously partici- 
pated in the survey in question. Breakouts are generally limited to 
two-year/four-year and public/private/ptoprietary classifications. 

There is alwi^s some concern that comparative institutional data 
will be improperly used, particularly in a normative sense. But valid 
reasons exist to justify and support the exchange of comparable 
student-outcomes data/Comparable data may illuminate problem 
areas within an institution while also showing that certain problems 
are common across institutions and are perhaps not correctable by 
individual schools. An institution using a graduate follow-up ques-^; 
tionnaire, for example, may fii^-that graduates in certain prbgram 
areas tiave ' difficulty ' obtaining jobs. Comparable data from 
graduating students at other institutions can show whether the 
problem is universal or is limited to a single institution. 

In sum, the concept of assembling data on students from different 
sources into a single ongoing data base and of using data drawn from 
that data base flexibly and comparatively to meet the challenge of 
particular administrative decisions as they arise cannot be over-, 
stressed. Data assembled and used in this way will have a consider- 
ably greater institutional impact than will a sitigle comprehensive 
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report issued after each data-gathering effort. Institutional adminis- 
trators seeking to maximize use of student-outcomes information 
should be fully aware of this. They also should encourage those 
responsible for collecting and documenting the results of student- 
outcomes studies to aggregate data in this manner. 

This admonishment leads to some closing observations on the 
appropriate relationship and division of labor between the adminis- 
trator and those responsible for collecting, aggregating, and analyz- 
ing.data on student outcomes. In his concluding discussion on the . 
role of the information professional in mediating between "data** 
and "information," Jones (1982; pp. 48-50) points out the dangers in 
■ allowing analysts, on the one hand, to determine policy questions 
and administrators trying, on the other handj to be their oWn 
analysts. In soliciting information, administrators should be fully 
aware of the analyst's tendency, in the absence of specific guidance, 
to treat all data elements as though they were of equivalent policy 
significance— a tendency that might be described as "the democratic 
theory of data." At the same time, administrators must listen closely 
and Gi'itically to what analysts have to say about the limitations and 
complexities of available data. Neither party can afford to assume 
that the other knows the problems and perspectives of his own role. 
Ultimately, it is the responsibility of the administrator to see that 
effective communication takes place. This is primarily because it is 
largely the administrator's misfortune if it does not take place. 
'. •.These general observations are particularly germane when the 
data ibeing considered are student-outcomes . data. Because of 
disciplinary training and classroom experiences, administrators 
generally >yiir know intuitively a great deal about student 
outcomes— more, probably, than about costs, buildings, and the 
other kinds of things they routinely must make decisions about. 
This means that they will often demand a. great deal of student- 
outcomes information and be disappointed when clear.answers are 
not forthcoming. This disappointment may be reinforced by the 
analyst who sees in student-outcomes data many methodological 
flaws and consequently qualifies. results to an unusual degree. Most 
studentoutijomes data simply are not heat. Many analysts do not 
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like to deal with such data, and most assume thi-t administrators 
should not want to either. Consequently, administrators must^take 
particular care to let analysts know that even imperfect information 
will be of considerable value, and to enlist the analyst's support- in 
trying to uncover as many indicators as possible of what is, after all, 
from a measurement standpoint, an unusually elusive set of 
phenomena. 



CHAPTER 4 



Applying 
Student-Outcomes 

Information to 

Institutiomd 
Decisionmaking 



Ensuring that data on student outcomes,, once collected, are 
/regularly used to effect positive changes is a twofold concern. 
First, careful attention must be given to the kinds of established 
' institutional.processes into which information on student outcomes 
can be easily, introduced. Second, a new consciousness must be 
fostered among administrators that .student success is a priority 
management goal. This chapter outlines briefly some typical uses of 
outcomes information the author has observed at various colleges 
and universities. It also presents some important themes that seem 
td characterize successful institutional effort to use student- 
outcomes information. ; 

Not every information-utilization effort will cause widespread 
change, of course, nor should it. Student questionnaire responses 
on a particular issue, for example, may be only the first information 
available on a -potential problem; decisionmakers will want to 
^ assemble information from other sources to see if any change is 
\ warranted. Indeed, a number of institutions use, the results of 
\studentKDUtcomes surveys in precisely this manner— to help pin- 
point potential student problems that can then be made the subject 
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of more intensive investigations. These investigations may go 
entirely . beyond studentoutcomes information. In this way, 
student-outcomes assessments, regularly conducted, can help set an 
effective institutional research agenda. 

The point has been made several times in earlier discussions that 
studentoutcomes information is often of much greater use in setting 
a context for administrati^'« decisionmaking than in informing par- 
ticular decisions. Increaser. use of student;outcomes information 
often leads to changes in the way certain kinds of decisions are 
approached— ir. the kinds of alterriatives considered, for example — 
rather than changes in the substance of decisions. . 



The Role of Student-Outcomes Information in 

Som e Common Institutional Activities 

■ One way of increasing the likelihood that student-outcomes infor- 
mation will be used by administrators is to put the information in a 
form suited to some of their regular activities. For most decision- 
makers, student-outcomes information falls into the category of^ 
"nice to know" rather than "need to know." Outcomes information 
is much more likely to be recognized as relevant if it is not seen as 
distinct from the kinds of "productivity" information upon \yhich 
most decisionmakers claim to base decisions. In fact, it should be 
both acknowledged, and stressed by top administrators that 
information about student outcomes represents an important 
component of any assessment of institutional or departmental/unit 
productivity. 

Student-outcomes information of all kinds has proved of value in 
a wide range of ongoing institutional activities and decision 
processes, such as the following: 

In the Preparation of Accreditation Self-Studies ^ 

'Uegional accreditation bodies are increasingly interested in assess- 
ing the degree to which institutional activities meet student needs. 
In addition, graduate surveys to evaluate student success in job 
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placement and in preparation for 'further study have long been 
'encouraged as a part of institutional self-studies. Studies of current 
students to assess satisfactibns and dissatisfactions with particular 
aspects of the college experience are also being encouraged. 'Indeed, 
some accrediting bodies have' begun to tie evidence of outcomes 
performance to accreditation standards (Southern Association of 
Colleges and Schools 1982). The kinds of information provided by 
student-outcomes surveys and assembled in an accessible, well- 
documented data base are of much help in meeting these require- 
ments, particularly if the information has been collected on a 
regular basis. One large northeastern public university recendy 
made a description of the structure and findings of its ongoing 
information-gathering processes the centerpiece of its accreditation 
report. Prominence was given -to the collection and analysis of 
student-outcomes information. 

In Program Review/Unit Evaluation Activities 

Among public institutions particularly, academic program review 
has been growing in importance as governing boards seek ways to 
allocate scarce resources to the most, effective programs. A similar 
trend has been apparent among private institutions. In the past, 
academ^ic program reviews in . the past were concerned primarily 
with an assessment of faculty quality and the ran^aand depth of the 
curriculum. Now the tende'ncy is fo include an^utcomes compo- 
nent. Usually this is iri the form of a survey of program graduates to 
assess preparation for employment and for graduate education in 
the field ofthe program. Student-outcomes surveys have substantial 
advantages in providing information to aid this process. By using 
different sets of departmentally designed local questions on stan- 
dard survey instruments, the institution can obtain data to enable 
interdepartmental comparisons to be made, while allowing each 
department to investigate the issues that concern it most. Similarly, 
reviews of nonacademic or service-unit activities are increasingly 
requiring a user survey. Questions on levels of use and satisfaction 
with services that appear on most student-outpomes surveys address 
these needs directly among different populations of students. Again, 
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it should be emphasized that such evaluations are more valid and | 
valuable if collected regularly, rather than on an ad hoc basis by J 
individual units. ^ ' ■ 

In Institutional Planning and Budget Review Activities / • ^ .\ 

The uses of student-outcomes information in-institutional plan* 
ning are numerous. Outcomes data may be added to more tradi' 
tional kinds of unit-productivity data to support unit allocation 
decisions, or to point to the need for special programs of services. At 
the same time, outcomes data coupled with needs'assessrnent dat^l 
in a particular plrogram area can do much to ensure that plafined ; 
new programs are aimed at the proper rnarket or regional consti- 
tuency. Finally, outcomes data can be used in their widest scope— to : 
.give (College and 'university, leadership insight into the degree to 
which establisBed "and agreed-upon elements' of • institutional ■ 
mission are being attained; One small private liberal-arts college is 
using various kinjis of outcomes infi^^jmati^gprto review its mission in V 
the light of student characteristics and achievements. A regional 
public institution is using graduate-survey resuks together with 
regional economic projections to prepare a schedule for planning 
and implementing new occupational programs. And a§^ final 
example, a large urban community college is currently developing a: 
computer-based, comprehensive, outcomes-oriented planning 
system to provide a context for annual budget reviews. 

In Developing Student-Retention Strategies • 

Student-outcomes surveys will often'reveal patterns of difficulties * 
and^dissatisfaction among different kinds* of current students— 
patterns that mky lead to a decision to leave the institution. ^ 
Comparisons of the responses of graduates apd former students may 
also reveal considerable differences in academic and personal gpalsj 
as well as different perceptions of and* satisfaction with- the institu- 
tion. Bqth kinds of information are highly useful in planning and 
implenknting institutional retention stratiegies. A number of insti- 1:: 
tutiotis are' using student-outcomes survey results in conjunction^;: 



INSTITUTIONAL DECISIONMAKING 



with other student data on dropouts to try to. isolate a set of early 
warning signals of a student's impending decision to leave the insti- 
tution. High-risk students can thus be identified at an early stage 
and be given special attention. Similarly, many institutions are using 
student-outcomes surveys to determine the levels of use of particular 
student-service areas by dropouts and-by persisters. If dropouts are 
not aware of, are not using, or are not satisfied with a particular 
service, efforts are made to improve its visibility and the dimensions • 
of service provided. 

Using student-outcomes ' data to develop student-retention 
programs should be an institutionwide activity. Frequently the 
feeling is present on campus that retention is a student-services 
problem. In fact, most student-outcomes research has shown 
program quality and faculty contact to be among the major factors 
involved in retention (see Terenzini 1979). Research also has shown 
that the most effective-retention strategies are institutionwide and 
have been developed with the full participation of all institutional 
constituencies— faculty, staff, students, and administration (Beal 
and Noel 1980). Once more, the importance of a committee with 
broad representation from all constituencies to review the retention 
implications of student-outcomes information cannot be over- 
stressed. One public regional university has. formally established 
such a committee in its governance structure, with the responsibility 
to make retention recommendations to the president on a regular 
basis. Reviewing and discussing the implications of a wide variety of 
student-outcomes information is one of this committee's primary 
tasks. 

In Developing Recruitment Materials and Strategies 

Outcomes information can document student successes and 
achievements. Knowing the characteristics of the persisting, suc- 
cessful student can help determine what kinds of students should be 
given priority in recruitment. Furthermore, effective presentation of 
the success of recent graduates jn findingemploymerit or in further- 
ing their professional development can increase interest in the insti- 
tution among high-quality students. On the other hand, identifying 
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the special problems of particular student populations— minority 
students, older students or part-time students, for example— may 
enable the institution to develop targeted recruitment materials. 
Such materials may be designed to help prospective students from 
these groups more effectively prepare themselves for college work. 
Or they may assure prospective students that appropriate support 
services to meet their needs are available. A large southeastern 
public university, for example, has recently developed a minority- 
recruitment strategy abased upon reviewing the success of its 
minority graduates and upon stressing the kinds of high-school 
preparation required to perform effectively at the university. 

These are but a few examples of the numerous kinds of insti- 
tutional problems or processes to which student-outcomes informa- 
tion can be effectively applied. 



Some Themes of Information Utilization 



A number of research or demonstration efforts have recently been 
undertaken on the use of student-outcomes information in institu- 
tional decisionmaking. Among these is a three-ryear NCHEMS 
demonstration project, funded by the W. K. Kellogg Foundation, 
with sites at seven public universities and community colleges 
nationwide. The evidence developing from this and other projects 
on uses of student-outcomes information indicates that such infor- 
mation is likely to be used by administrators only when a number of. 
factors are operating to positively orient the entire campus to the use 
of information. It appears that information use will be most success- 
ful when the following themes are present: 

• When the information needs of institutional decisionmakers are 
carefully assessed by those collecting the data, and the relevance of 
student-outcomes information to their particular area of responsi- 
bility is stressed continually by those providing it. This is a crucial 
point. Most bodies of assessment information sit on shelves, 
unused largely for two reasons: they are not communicated to 
managers in the language of management, and they do not 
draw direct relationships between obtained results and 
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current management problems. In overcoming this difficulty, 
there.is no substitute for explicit, face-to-facecommunications. 
A number of tactics have proved helpflil in actual college and 
university settings. These include surveys of awareness on 
information availability and utility, workshops on available 
information and its implications, inclusion of nontechnical 
administrators on survey- and study-design committees, and 
evaluations of the relevance of current data-distribution . 
methods. 

When the information collected is disseminated in a series of tailored 
memos to particular administrators or is included in the response to a, 
partiadar decisionmaker's request for information. This point 
follows from the previous theme. It emphasizes the need to 
establish a consistently structured institutional student- 
outcomes data base to which new data maybe added regularly. 
The inclination of institutional researchers to report each 
data-gathering effort as an independent entity should be sum- 
marily discouraged, since such research reports are rarely used 
by practicing administrators. Rather, researchers should be 
encouraged to regularly report on the kinds of data available, 
and the kinds of uses administrators might find for newly 
available data. Finally, the promptness with which a given 
administrator's request for information on a particular 
problem is met is probably the greatest determinant of 
whether another such request will be. forthcoming. 
When the information collected is presented comparatively so that 
contrasts between different types of students and trends over time are 
highlighted— often through graphic presentation. Perhaps nothing 
is more easily accomplished and less often practiced than the 
effective comparative presentation of data. The trend revealed 
by a nicely laid put comparative graph or table conveys much 
more^ useful information to. an administrator than a 'more 
detailed presentation of facts collected at. one point in time 
'(Hackm:an 1983). The same can be said, of course, about com- 
parisons among institutions or among departments or units 
within the same institution, provided appropriate caveats 
about differences in institutional mission and structure are 
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heeded. It should be kept in mind, too, that outcomes data are 
usually gathered by means of a questionnaire survey. Because 
of the many potentially biasing factors present in such data, 
relative upward or downward trends in a particular data 
element are likely to be much more valid than the value of a 
given measurement per se. 

When the information collected is effectively integrated u)ith other . 
kinds of student data—often more familiar to the decisionmaker--t6 
yield a comprehensive picture of a particular problem. Once again, 
most administrators tend to class student-outcomes informa- 
tion as contextual or background information— not imme- 
diately useful in the solution of a particular current problem. 
Only when outcomes data is placed alongside more traditional 
indicators of productivity and effectiveness will administrators 
recognize its utility in assessing unit operations. And note that 
alongside means on the same piece of paper or in the same table 
shell. A single column of data reporting the percentage of 
graduates applying to and accepted by graduate schools 
should be placed directly opposite columns of data showing 
departmental costs and loads. This will be far more effective in 
promoting outcomes-information use institutionwide than b11 
of the rhetoric in the world. 

When a clearly identifiable institutional problem is present, for 
example, a high attrition rate-^and where there is highAevel admin^ 
istrative commitment to solve the problem. Increasing information 
use among administrators often involves changes in funda- 
mental attitudes, and attitude change rarely occurs unless a 
highly salient problem is present.- Most successful institutional 
efforts to. more effectively utilize information on student out- 
comes have relied heavily on the widespread recognition that a 
problem exists and that the problem is of sufficient magnitude 
that new approaches to thinking about management are 
worth trying. In some cases, these approaches reveal that the 
size or nature of the problem itself has been misestimated. In 
dny case, new persisting habits of information use may be 
established. ' 
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When an appropriate fonm {or meaningfully disaissing the institu- 
tional implications of stxulent-oxacomes information is present^-for 
example, a retention or student-success committee with broad ' 
representation and high-level administrative endorsement. The 
value of such committees has been underscored throughout 
this book, for a number of reasons. Most important, perhaps, 
is the fact that a single forum for discussing issues of student 
success and failure is almost never present in higher-education 
institutions. Moreover, the very structure of most institutions 
tends to preclude such discussion on a regular basis. Adminis- 
trators thus seldom have the opportunity to view the institu^ 
tion as the student sees it— as a total environment composed of 
many interlocking parts. Putting available student-outcomes 
data before a committee whose members are drawn from 
throughout the institution is probably the best way to make 
sure that all sides of the student experience are covered, and 
taken into account when analyzing the data (Astin 1976). 
Such committees are often composed of decisionmakers in 
their own right— individuals able to make immediate changes 
in their own units if the data seem to imply that changes are 
warranted. Such changes can take place regardless of the 
content or outcome of committee discussion. Finally, of 
course, committee structures are part of the common flora and 
fauna of university life. In spite of the standard reaction, "not 
another committee,'* such bodies do symbolize administrative 
concern and institutionalize efforts that otherwise would be 
extremely difficult to focus in a complex, decentralized 
environment. 

• Final!)', when an attitude can be developed throughout the institu^ 
tion that the improvement of student outcomes is important, can be 
accomplished^ and will be rewarded. None of the above themes is 

. as critical to success as a clear and visible commitment by 
top administrators to making faculty and unit heads more ' 
accountable for student success. Student-outcomes informa- 
tion is not generally accorded a salient position in the minds of 
most administrators because giving it attention is not a factor 
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in the institutional reward structure. As a result, cxecucive-- 
level deoisionmakers have to recognize that changes will not 
occur unless appropriate incentives for change are provided 
and consistently delivered when innovations do occur. 



The Need for Communication 



Development of the conditions described above at a given 
institution— particularly the last theme- -is bound to be a slov 
process. In the ultimate analysis, using student-outcomes informa- 
tion requires a change of attitude on the part of most administrators. 
They must 'come to realize tl^at this kind of information can help 
solve their problem. It is up to those responsible for collecting data 
and communicating information to help foster this change. What 
work best ave prompt, short, problem-specific responses to par- 
ticular concerns, along with encouragement of information users to 
communicate \^'hat they need and to critically evaluate what they 
have been given. But equally it is up to administrators themselves to 
ask for information in this manner anci to give appropriate feedback 
to institutional data-collection personnel. 

A significant obstacle to the development of effective, informed 
student-success programs is the structure of colleges and universities 
themselves. Student persistence and i\chieyem^:nt are not generally 
j.^ttributable to any one feature or program of the institution. 
Instvad, they are the product of a- complex set of factors, working 
together and cutting across all aspects of college and university life. 
Individual facuKy members and administrators, however, do not 
usually deal with more than a single ai>pect of a given student*s 
involvement in the institution— as a student in a particular class, as 
a candidate for financial aid, as an admissions exception seeking 
additional help i n developing studv skills, and so on. Furthermore, 
precisely because student success is rarely attributable to a single 
cffieeor function, assessments of individual unit oc program success 
are most easily (and probably appropriately) directed at the contents 
of the services delivered rather than at the effects these services 
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produce. Because of its holistic nature, student success is 
everybody's business but not anybody's specific responsibility. 

Dealing, with this situation at any institution requires at least two 
kinds of initiatives— neither directly related to the amount and 
quality of information on student outcomes available. First, it 
requires a plain and visible commitment by top administration that 
the issue of improving student outcomes is a priority and that unit 
initiatives consistent with this priority will be rewarded. So long as 
the standard of managerial accountability remains efficiency rather 
than effectiveness, the likelihood of student-outcomes information 
being utilized, regardless of its quality, remains low. 

A second requirement is coordination and communication 
among the various efforts undertaken as a result of such a commit' 
ment. This requirement is perhaps most significant when mobilizing 
data resources. Each unit head must not only understand the goals 
of contemplated student-success programs but should also under- 
stand the potentially reinforcing or conflicting interrelations among 
the new policies affecting different units. Finally, the different units 
involved in achieving student-success goals must frequently share 
information— not only on what they are doing and plan to do, but 
also on what they have learned individually about the factors deter- 
mining successful outcomes for different kinds of students. 

If individual units and administrators are held definitely account- 
able For student outcomes, and if incentives are created for lateral 
information-sharing among units, the picture for effective utiliza- 
tion of student outcomes is rosy indeed. The dark side of this picture 
is that student-outcomes information has the potential to become 
extremely political as its use in decisionmaking increases. This has 
Jong been true of output and productivity data. Efforts to include 
outcomes assessments in institutional program-review processes, for 
example, have been seen by some unit heads (and^by'tRe majority of 
faculty) as part of a veiled agenda on the part of qentral administra- 
tion to establish greater control over budgets (Barak 1982). And it 
has to be said that in the ma:jority of cases, these fears have not 
proved to be totally groundless. 
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Such fears have recently become even greater, and perhaps with 
greater cause, when comparing outcomes across institutions is 
proposed. In the public sector, at least one state is now allocating a 
small portion of its higher-education dolldrs on the basis of 
demonstrated institutional performance (Bogue 1982). Such efforts 
are indeed admirable to the extent that they signify commitment to 
student success, which they both reveal and institutionalize. They 
are less admirable if they begin to'blur legitimate differences in what 
different kinds of institutions are tryirig to accomplish-differences ^ 
based upon mission, academic tradition, or types of students 
enrolled. These issues will be more flilly discussed in the final 
chapter. 
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A Concluding 
Comment 



The preceding chapter pointed up the fact that establishing the 
appropriate role of information in administrative decision- 

• making calls at least as much for sound intuition as it does for precise 
method. Nowhere is this more true than in the realm of assessing 
and interpreting student outcomes. In chapter 2, we saw that this is 
partly due to the nature of outcomes themselves— particularly that 
they are complex, multifaceted, and multicausal. And in chapter 3, 
we sa\y that intuitive judgment is a key factor because of the many 
technicaMimitations associated with most sources of .data on 
student outcomes— particularly because they are. often imprecise 
and fragmentary. At the outset, the point was made that these two 

, kinds of inherent limitations on student-outcomes data have led 
many in higher' education to conclude that any attempt to assess 

* educational outcomes is at best questionable and at worst 
. dangerous. Those who maintain this position, however, ignore the 

fact that in the absence of such measures, other criteria, far more 
questionable and dangerous, most certainly will be applied. 

There is nothing new, nothing revolutionary, about assessing and 
placing value on the products of a particular educational experience. , 
Indeed, one need go no further than a class grade list to make the 

59 

63 



EKLC 



Peter EwcU 



point that outcomes judgments arc both ubiquitous and legitimate 
in •higher education. Both faculty and administrators routinely make 
juclgments about program effectiveness— and make changes as n 
result of those judgments. What is new and somewhat threatening 
about the kinds of outcomes assessments we have looked at in this 
book is that they are more explicit than more conventional modes. 
That is, they provide more evidence,' in more detail, about more 
aspects of the educational experience, and they do so in a fashion 
that allows direct, measurement^based comparisons to be made 
among, programs and among Institutions. Because the nature of 
these differences is often not well understood, some fijrther discus, 
sion of these larger issues seems warranted. 

Many who take exception to explicit assessment of student 
outcomes confuse explicit with nonsubjective. The case for the sup- 
position tliat most educational outcomes cannot be measured rests 
heavily c/n the assumption that measurement always involve? an 
attempt |o reduce the content of a rich and varied experience to a 
. qualified objective standard, seemingly lacking in such qualities. As 
chaptcri 2 and 3 have indicated, however, the matter is considerably 
more cbmplex. In fact, the bulk of current outcomes-assessment 
instruments and techniques remain subjective in the ser je th^t they 
are based on informed reactions to the educational process of those 
closest to it--students and former students. Interpretation and use . 
of such data, as we noted in chapter 3, is subject to its own set of 
limitations. If collected during or soon after the experience, self- 
ratings willoften reflect situational factors affecting the experience. 
Therefore, they may notililly indicate levels ofgrowth-particularly 
in educational experiences involving considerable challenge and dif- 
ficulty. If self-ratings are collected long after the experience, growth 
' may similarly be exaggerated or misattributed, because of the 
human tendency to romance during recollection, in either instance, 
outcomes data based on self-ratings are rightly treated with some 
caution. 

- But the persistent widespread use of self-ratings also testifies to 
their utility. Such data have an in\mediacy and an applicability not 
typicatof more objective measures . It is hard to imagine an adminis- 
trator failing to investigate fijrther, upon receipt of a number of 
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responses from former studeius— clustered in n particular program- 
to the effect that they had gained litde or nothing from their course 
of study. It is equally hard to imagine administrators keeping to 
themselves the i<novvledge that substantial majorities of graduates 
value their undergraduate experience highly and attribute to it a 
generous portion of their subsequent occupational achievement. 
Both types of findings have important implications for action both 
inside and outside tbe institution, yet neither is derived from what 
would be called an objccdve» or externally verified, assessment 
technique. More important, neitber would be discovered at all if 
explicit outcomes assessment had been rejected in principle from the 
outset. ■ 

Consider, too, that most techniques and instruments for assessing . 
stLjdent outcomes that clo not involve self-radngs arc equally based V 
on informed subjective jj^dgipents. A major advantage of the, 
American College Tcsnng Program'*s College Outcome Measures 
Project (COMP)» for example, is. that important components of the . 
assessment rely heavily on individual faculty judgments of student 
performance. And one of the most valued kinds of outcomes data, if 
least often collected, consists of employer ratings of student prepara- 
tion for the field in which they were trained. Even much-maligried 
staiidardized achievement. tests, when properly considered, consist 
of no more , than the previously recorded judgments of a panel of 
scholars in a particular discipline. C. Robert Pace recendy observed 
(1983) from a somewhat different perspective, 

The opinions and judgments of faculty members' and of 
students are based upon a great deal of observation and experi- 
ence. One of the great advertising slogans^of all time was for 
the elegant Packard automobile in the 1920*s: *Ask the man. 
who Owns one.* [P. 16] ' . . 

While not objecdve in the sense of involving a physical measure- 
ment, such assessments Kave the consequential property of being 
most, mtersubjective. This is, different ''observers operating unde'r 
different conditions at different times will generally agree, on the 
properties of a given outcome and will be in substantial agreement as 
to its value. This presents a considerable advantage over subjective 
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Judgments mndc by one individiinl, such as grades on the one hnnd 
nnd student self-rntlngs on the other, A substantial internal 
advantage of intersubjcctive assessments in the management of a 
particular program or in^'titution is that they can be fairly reliably 
compared with one another. They also are likely to be a good deal 
more credible to those outside the higher-education community 
thnn are traditional subjective assessments, it could be fairly argued 
that higher education's greatest challenge with respect to outcomes, 
assessment lies not in the actual mcmiremcnt of outcomes, it lies 
rather in achieving consensus on which outcomes ought to be 
assessed and what kinds of outcornes should be taken as indications 
of institutional success. Only in thisAvay can e//ecnVcne55 emerge as 
the paramount criterion for success in higher educadon. 

This consensual element in most studerit-outcomes assessments is 

often obscured by the fact that assessment result.s generally are 
expressed in quantitadve terms. Indeed, many of those most vocally 
opposed to explicit student^outcomes assessment are reacting more 
to the numeric packaging of assessment results than to their actual 
content or implications. Again, it is ironic that individual course 
grades are less often subjected to the same criticisms. In themselves, 
numeric scores and letter grades are neither dehumanizing nor 
precise. Both represent the encoded judgments of one or more 
external human observers, and each is useful insofar as it allows 
meaningful distinctions to be drawn among different populadons 
regarding what they have achieved or experienced. Mostimportant 
of all, each is only an indicator of the occurrence of a particular out- 
come: it is not, and does not ' .'et nd to be, the outcome itself. 

This indicative quality of lU^-i; student-outcomes research is 
probably the aspect least well understood by its critics. As we have 
seen, most procedures for gathering information on student out- 
comes are indirect and will provide only pardal information on*a 
given outcome. Informadon gathered in this manner is ordinarily 
much more useful for the questions that it raises than for the 
ahsWct-s it provides. One of the most widespread, successful, and 
appropriate uses of student'outcomes information is simply to focus 
administrative attendon on a potendal problem. If placement rates 
are found to be low in electrical engineering at a particular regional 
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university, tlint statistic tells an ncademic ndmlnlstrntor nlmost 
nothing about whether the cnuse is low program quality, Inndcqunte 
counseling,' or simply n Inck of avnllnble jobs In the region, The 
placement data are very likely, however, to prompt the adminis* 

. trator to raise further questions about that particular program. The 
nature of outcomes research is always such that we are looking at the 
shadows of things rather than the things themselves, and very early 
on we were taught not to judge things by the size of their shadows* 
On encountering a particularly large shadow, however, most of us 
have sense enough to look up quickly, 

An additional property of all indicators is that they are most 
useful when there are more than.onc of them. Together, different 
kinds of measures of the same outcome dimqnsion undoubtedly 
provide a fuller picture of the dynamics of a particular educational 
experience. What is more important, they give the observer confi- 
dence that something real is being measured. In fact, a major danger 
inherent in the way institutions and systems of higher education 
have recently approached outcomes measurement is. the reliance* 
placed on single indicators of student performance^ As efficiency 
criteria have been given increasing weight in institutional assess; 
ment, the institutions have discovered a variety of cleverly devious 
ways to measure the conceptually straightforward notion of instruc- 
tional costs. The resultant problems associated with statewide cost 
studies alone. should be sufficient warning to those attempting to 
implement assessment schemes for instructional effectiveness based 
on a single criterion or indicator. Such efforts have been relatively 
few to date.;But they are bound to multiply as discussions of and 
concern about instructional effectiveness increase. 

Consideration of the dangers inherent in judgmental, single- 
indicator approaches leads naturally to a discussion of the compara- 
tive use of outcomes data. On the face of it, there is little point in 
collecting outcomes data at all if they are not used comparatively. A 
priority objective of most explicit outcomes-assessment efforts is to 
generate comparable information. The Intent is to use the informa; 
tion to identify the relative strengths and weaknesses of particular 

• instructional programs, or the relative difficulties or successes 
experienced by different populations of students. Furthermore, 
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hcciuisc of the luuurc of most studctu-outcomcs data, informutioii 
on the relaiwc stmulinK of pnrticulnr prosmms or bodies of students 
with re^^ard to a ^ivcn outcome nKMisurc is usually considerably 
more valid and reliable than the absolute scores ihemselyes. Take as 
an example an lutitudinal item on a student follow-up questionnaire 
seeking ratings of perceived gain in n studcnt^s major field., 
Responses iiMiy reveal considerable differences between the 
graduates of two^Kograms. Changing a single word in the text of the 
questionnaire item may have significant impact on the assessment 
scores of both prograiiis' graduates but is nonetheless likely to 
preserve the difference between them. In a few rare cases— graduate 
salary data, placernent rates, and performance in graduate study, for 
instance— absoKitc measures of outcomes may have substantive 
mei^ning. But in the vast, majority of cases, it is only through 
comparative analysis of relative performance that outcomes data 
acquire value to institutional administrators. 

Nevertheless, ;he potential for abuse of outcomes data employed 
in comparisons is considerable— particularly when comparisons are 
made across institutions. The risk escalates when those making the 
comparisons are remote fr6m and unfamiliar with the higher* 
education community. An initial caution here is elementary: not all 
institutions of higher learning are |fying to accomplish the same 
thing. Community colleges have longlaborcd under the charge that 
they are ineffective because only a smefll proportion of each entering 
class attains a degree. Yet degree attainment usually is a highly 
inappropriate indicator of effectiveness given the mission and 
programmatic structure of most community colleges. Similarly, 
salary on the first job after separation is an inappropriate indicator 
of the effectiveness of a four-year liberal-arts curriculuVny-al^^o^gb 
community colleges might welcome assessment in terms of such an 
•indicator. 

Institutions differ not only in what they are trying to accomplish 
but in the kinds and qualities of students they serve. Much of the 
heat in the current debate about assessment of "value added** is 
generated by this issue. Is it more appropriate to evaluate institu- 
tions comparatively in terms of what their students can do on 
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exit — regardless of the content of their educational experience? Or 
should they be compared in terms of what their students can do 

■ differently or better — regardless of their level of absolute performance 
(Astin 1982; Manning 1982)? Quite apart from the merits of either 
approach, it is clear that the differences between the institutional 
rankings produced by each mode of comparison would be profound. 

Because of inherent differences in institutional mission and in the 
kinds of students enrolled, 'direct comparisons among institutions 
involving most of the available outcomes measures should be 
approached with considerable caution. If an appropriate peer group 
is chosen, comparative studies may well be of value in pointing out 
institutional strengths and weaknesses. But such analyses will 
always be more helpful if they are approached in an exploratory 
rather than judgmental manner. That is, they should be valued 
rhore for the questions they raise than for the frequently invidious 
contrasts they suggest (Lawrence 1982). 

More difficulty arises from the fact that few meaningful outcomes 
measures are available across institutions. Even ^uch simple indica- 
tors as graduation and placement rates are far from being uhifornily 
collected. And in view of the variety of, attitudinal assessment 
instruments in general use, interinstitutional comparisons present a 
formidable methodological challenge, quite aside from their concep- 

; tual difficulties. Most of the major commercial outcomes assessment 
services publish national norms of results across institutions, and 

- these results can be quite useful if treated with appropriate caution. , 
They are limited, however, to institutions that have chosen to 
participate in the service in question- — very far from a random crpss 
section of institutions. 

Despite all of the difficulties, a number of institutions have found 
comparative analyses of outcomes to be valuable. This is especially 
so of institutions that in a given state have been the first to report 
them. One small midwestern regional state university, for example, 
has consistently attracted favorable attention froin its governing 
board and legislature by collecting and pubMiini' r<5:atistics on the 
comparative performance of its exiting studevir^i on nationally 
administered field-achievement and professional'Ct^nification tests. 
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This institution also has in place a value-added assessment pro-/ 
cedure that tests gains achieved in the first two years of enrollment. 
Other institutions maintain programs of this sort, in whole or in 
part. But a distinctive attribute of this institution's approach is that 
explicit criteria derived from the assessment program are made the 
concrete basis for funding requests to the board and legislature. The 
initiative for using data in this manner has remained with the insti- 
tution. Therefore, the kinds and variety of data used in the assess- 
ment are those best suited for evaluation of the institution's m^ission, 
and also are collected in sufficient variety and discussednn enough 
detail internally to be seen as real aids to institutional development. 
Institutions may not long enjoy the luxury of developing/ on their , 
own such an open, participatory process, / 

It is time to sum up. At the most general level, studentK>utcomes^ 
information represents both a powerful managerial resource foty 
institutional self-improvement and a powerful collective resource for ^ 
restoring higher educ^ition to its former priority in th^/public mind. 
By means of a thoughtful and participatory program of student- 
outcomes assessment, an institution can assess a ^wide range of 
programmatic impacts on its students and thus compare its" actual 
achievements with' its stated educational aspirations. Administra- 
tors have both the right and the responsibility tq: create account- 
ability structures for themselves, for faculty, and foy students as well, 
to ensure that educational outcomes most nearly approach the insti- 
tution's goals. At' the same timei by means of collective, honest; and 
unself-consciousl communication of the results of outcome 
assessments to the public and to those with funding authority, the 
general credibility of the "self-evident" benefits of higher education 
can be more firmly reestablished. As Stauffer/(1981) quite rightly 
points out, the kjind of public conHdence resulting fVom honest com- 
munication abdlut program quality is the most precious asset the 
higher-education community possesses. Administrators therefore 
should in turn e\cpect to be held acc^ntable by faculty, students, 
and former students to ensure that such cbmmufiication indeed 

takes place. \ / • ^ 

Both in the administration of individual colleges and universities 
and in the advancement of higher education as a whole, goal setting 

■ • V — i— — - — ; ; 
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and making maximal use of available resources to attain identified 
goals— whatever their content or variety— remain the premier 
management responsibilities. Durable vigilance is required to see to, 
it that all available management information— including infor- 
mation on student outcomes— is conceptualized, collected, and used 
to support instead of hinder effective and creative management 
practice. 



APPENDIX 



Comparisons of 
Data-Gathering 
Instruments on Van< 
Outcomes Dimensions 



The following chart assesses six commercially available and 
commonly used instruments for collecting data about student 
outcomes. The instruments are presented in terms of their relative 
coverage of 29 distinct types of student outcomes, arranged on four . 
basic dimensions — the classification scheme used in the NCHEMS 
Outcomes Measures and Procedures Handbook. The chart also indi- 
cates whether the outcome in question is measured directly by the 
instrument (one or more items devoted exclusively to its assessment) . 
or measured indirectly (an assessment is possible through inferential 
use of data from one or more items). The reader should be careful to 
note that breadth of coverage is not the only virtue, and that each 
instrument should be carefully evaluated on its own merits before a 
decision to adopt it is made. If used judiciously, however, the chart 
should give the interested administrator a place to start in evalu- 
ating the merits of the range of instruments avaflable. 
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Summary of Student-Outcomes Dimensions for 
Comparative Use 

A. Student Knowledge and Skills De\-elopment Outcomes 
A-1 Student development concerning breadth of knowledge 
A-2 Student development concerning depth of knowledge 

A-3 Student success in passing certification and licensing examinations 
A^ Areas and agents of student change during college 

B. Student Educational Career Development Outcomes 
B-1 Highest degree or certificate planned 

5-2 Students enrolled in an organized educational activity for no credits 

5-3 Program completers during a certain time period 

5-4 Program completers who entered as transfer students 

8-5 Degrees and certificates earned by an entering class of students 

B-6 Time to program completion for a graduating class 

B-7 Time to program completion for an entering class . 

B-8 Educational program dropouts 

B-9 Students seeking additional degrees and certificates 

B-10 Students working toward and receiving another degree or certificate 

B-ll Student ability to transfer credits . . 

B-1 2 Level of achievement of former students in another institution 

C. Student Educational Satisfaction Outcomes , 

C-1 Student satisfaction with overall educational expedience 
C-2 Student satisfaction with vocational preparation 
C-3 Student satisfaction with knowledge and skills in the humanities 
Student satisfaction with critical thinking ability 
....C'5 Student satisfaction with human relations skills 

D. ' gtudent Occupational Career Development Outcomes 

D'l Student success in obtaining first job 

D'2 Student success in obtaining preferred first job 

D-3 Occupational career choice 

D^ Job satisfaction 

D'5 First job earnings 

D-6 Annual total income of former students ■ 
D'7 Employment in major field of study 

D-S Change and stability of career goals ■ „ ^ ■ 



DATA^G ATHERING INSTRUMENTS 


A Comparison oif Six Outcome Instruments on 
Various Dimensions 
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O = Indirectly Measured 
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